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IN THE CLAIMS 

Please cancel claims 1-56 without prejudice. 
Please add the following new claims: 

1 57/ (New) A method implemented by a digital processing system for 

2 ^processing media data, said method comprising: 

3 retrieving from a digital storage system a set of data which 

4 Indicates how to transmit a time related sequence of media data according to a 

5 transmission protocol, wherein said set of data is a time related sequence of 

6 data associated with and separate from said time related sequence of media 

7 data. 



1 58. (New) A method as in claim 57 further comprising: 

2 transmitting packets of data representing said time related 

3 sequence of media data according to said transmission protocol. 

1 59. (New) A method as in claim 57 wherein for each of said packets. 



2 said set of data refers to data In at least one of a sequence of image data or a 

3 sequence of audio data associated with said time related sequence of media 

4 data. 
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1 60. (New) A method as in claim 57 wlierein said method further 

2 comprises packetizing said time related sequence of media data according to 

3 said set of data. 

1 61 . / (New) A machine readable medium containing executable 

2 omgram instructions, which when executed on a digital processing system 

3 cause the digital processing system to perform a method comprising: 

4 retrieving a set of data which indicates how to transmit a time 

5 related sequence of media data according to a transmission protocol wherein 

6 said set of data is a time related sequence of data associated with and separate 

7 from said time related sequence of media data. 

1 62. (New) The machine readable medium as in claim 61 , said method 

2 further comprising: 

3 transmitting data representative of said time related sequence of 

4 media data according to said set of data. 

1 63. (New) The machine readable medium of claim 61, wherein said 

2 set of data is stored as a track of indicating data. 
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1 

2 



64. (New) The machine readable medium as in claim 61 wherein said 

transmission protocol comprises a packet data protocol. 



1 65. (New) The machine readable medium of claim 61 , wherein said 

2 method further comprises: 

3 determining a format of said time related sequence of media data; 

4 packetizing said time related sequence of media data according to 

5 said set of data; 

6 wherein said transmission protocol is used to transmit said time 

7 related sequence of media data which has said format and wherein said 

8 packetizing uses said format and said protocol to packetize said time related 

9 sequence of media data. 



1 66. (New) The machine readable medium of claim 65, wherein the 

2 method further comprises: 

3 transmitting packets of data representing said time related 

4 sequence of media data according to said transmission protocol. 



1 67. (New) The machine readable medium of claim 66, wherein for 

2 each of said packets, said set of data refers to data in at least one of a sequence 
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3 of image data or a sequence of audio data associated witli said time related 

4 sequence of media data. 

1 68./ (New) An apparatus comprising: 

2 / a port configured to receive a set of data associated with 

3 ' transmission of a time related sequence of media data according to a 

4 transmission protocol, wherein said set of data is a time related sequence of 

5 data associated with and separate from said time related sequence of media 

6 data; 

7 a processing unit coupled to said port to receive said set of data, 

8 said processing unit packetizing said time related sequence of media data 

9 according to said set of data. 

1 69. (New) The apparatus of claim 68, further comprising a transmitter 

2 coupled to said processing unit, said transmitter for transmitting packets of data 

3 representing said time related sequence of media data according to said 

4 transmission protocol. 

1 70. (New) The apparatus of claim 69, wherein for each of said 

2 packets, said set of data refers to data in at least one of a sequence of image 
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3 data or a sequence of audio data associated witii said time related sequence of 

4 media data. 




(New) An apparatus for processing media data, said apparatus 

a means for retrieving a set of data which indicates how to transmit 
a time related sequence of media data according to a 
transmission protocol, wherein said set of data is a time 
related sequence of data associated with and separate from 
said time related sequence of media data; and 

a means for packetizing said time related sequence of media data 
according to said set of data. 



1 72. (New) The apparatus of claim 71 , further comprising: 

2 a means for transmitting packets of data representing said time 

3 related sequence of media data. 

1 73. (New) The apparatus of claim 72, wherein for each of said 

2 packets, said set of data refers to data in at least one of a sequence of image 

3 data or a sequence of audio data associated with said time related sequence of 

4 media data. 
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1 74. / (New) A method implemented by a digital processing system for 

2 p^cessi^g media data, said method comprising: 

3 retrieving a first time related sequence of data to indicate how to 

4 transmit a second time related sequence of data according to a transmission 

5 protocol, wherein said second time related sequence of data is associated with 

6 time-based media, and wherein said first time related sequence of data is 

7 associated with said second time related sequence of data; and 

8 packetizing said second time related sequence of data according 

9 to said first time related sequence of data. 

1 75. (New) A method as in claim 74, further comprising: 

2 transmitting packets of data representing said second time related 

3 sequence of data according to said transmission protocol. 

1 76. (New) A method as in claim 75, wherein for each of said packets, 

2 said first time related sequence of data refers to at least one of a sequence of 

3 image data or a sequence of audio data associated with said second time 

4 related sequence of data. 
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1 77. (New) A method as in claim 76, wherein said second time related 

2 sequence of data is stored on a read-only memory (ROM). 

1 78/ (New) A machine readable medium containing executable 

2 ^ogram instructions, which when executed on a digital processing system 

3 cause the digital processing system to perform a method comprising: 

4 retrieving a set of data which indicates how to transmit a time 

5 related sequence of media data according to a transmission protocol wherein 

6 said set of data is a time related sequence of data associated with said time 

7 related sequence of media data. 

1 79. (New) The machine readable medium as in claim 78, said method 

2 further comprising: 

3 transmitting data representative of said time related sequence of 

4 media data according to said set of data. 

1 80. (New) The machine readable medium of claim 78, wherein said 

2 set of data is stored as a track of indicating data. 
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1 81 . (New) The machine readable medium as in claim 78 wherein said 

2 transmission protocol comprises a packet data protocol. 

1 82. (New) The machine readable medium of claim 78, wherein said 

2 method further comprises: 

3 determining a format of said time related sequence of media data; 

4 packetizing said time related sequence of media data according to 

5 said set of data; 

6 wherein said transmission protocol is used to transmit said time 

7 related sequence of media data which has said format and wherein said 

8 packetizing uses said format and said protocol to packetize said time related 

9 sequence of media data. 

1 83. (New) The machine readable medium of claim 82, wherein the 

2 method further comprises: 

3 transmitting packets of data representing said time related 

4 sequence of media data according to said transmission protocol. 

1 84. (New) The machine readable medium of claim 83, wherein for 

2 each of said packets, said set of data refers to data in at least one of a sequence 
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3 of image data or a sequence of audio data associated witli said time related 

4 sequence of media data. 

REMARKS 

Tfiis application is being filed as a divisional of application serial 
number 09/140,173, filed August 25, 1998. Claims pending in the instant 
application are numbered 57-84. Applicants respectfully request consideration 
of the instant divisional application as amended. 

If there are any additional charges, please charge Deposit 
Account No. 02-2666. 



Respectfully submitted, 



Blakely, Sokoloff, Taylor & Zafman 




(James C. Scheller, Jr. 
Reg. No. 31,195 



12400 Wilshire Boulevard 
Seventh Floor 

Los Angeles, California 90025-1026 
(408) 720-8598 



4860.P2207XD 

Filed: December 23, 1999 



- 10- 



Examiner: To be determined 
Art Unit: To be determined 



04860.P2207 



United States Patent application 

FOR 



METHOD AND APPARATUS FOR MEDIA DATA TRANSMISSION 



INVENTORS: 

ANNE JONES 
JAYGEAGAN 

Kevin L. Gong 

ALAGU PERIYANNAN 

David W. Singer 



PREPARED BY: 

BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN 
12400 WILSHIRE BOULEVARD 

SEVENTH FLOOR 
LOS ANGELES, CA 90025-1026 

(408) 720-8598 



FXPWFSS MAn. TFRTmCATK OF MAILING 
"Express Mail" mailing label mimha- F.M492088673US 



Date of Deposit: August 25. 1998 



I hereby certify that I am causing this paper or fee to be deposited with the United States Postal Service 
"Express Mail Post Office to Addressee" service on the date indicated above and that this paper or fee has 
been addressed to the Assistant Commissioner for Patents, Washington, D. C. 20231 



Connie Thaver 



(Typed or printed name of person mailing paper or fee) 



(Signature of person mailing paper or fee) 



(Date signed) 



METHOD AND APPARATUS FOR MEDIA DATA TRANSMISSION 

FIELD OF THE INVENTION 
5 The present invention relates to methods and apparatuses for preparing time 

related sequences of media data for transmission, and more particularly to packetized 
transmission of such media data. 

INTRODUCTION AND BACKGROUND 

10 There are various different file structures used today to store time-based media: 

audio formats such as AIFF, video formats such as AVI, and streaming formats such 
as RealMedia. One reason that such file structures are different is their different focus 
and applicability. Some of these formats are sufficiently relatively widely accepted, 
broad in their application, and somewhat simple to implement, and thus, may be used 

15 not only for content delivery but also as interchange formats. Foremost among these 
general foraiats is the QuickTime file format. It is used today in the majority of web 
sites serving time-based data; in the majority of authoring environments, including 
professional ones; and on the majority of multimedia CDROM tities. 

The QuickTime media layer supports the efficient display and management of 

20 general multimedia data, with an emphasis on time-based material (video, audio, etc.). 
The media layer uses the QuickTime file format as the storage and interchange format 
for media information. The architectural capabilities of the layer are generally broader 
than the existing implementations, and the file format is capable of representing more 
information tiian is currentiy demanded by the existing QuickTime implementations. 



/ 
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In contrast to formats such as AVI, which were generally designed to support 
local random access of synchronized media, QuickTime allows systems to manage the 
data, relationships and timing of a general multimedia presentation. In particular, the 
QuickTime file format has structures to represent the temporal behavior of general 
time-based streams, a concept which covers the time-based emission of network 
packets, as well as the time-based local presentation of multimedia data. 

The existing QuickTime file format is publicly described by Apple Computer 
in the May 1996 File format specification, which may be found at the QuickTime site, 
<http:/Awww.applexom/quicktime>. 

One aspect of the QuickTime file format is the concept that the physical 
structure of media data (the layout in disk records) is independent of, and described 
by, a logical structure for the file. The file is fully described by a set of "movie" meta- 
data. This meta-data provides declarative, structural and temporal information about 
the actual media data. 

The media data may be in the same file as the description data, (the "movie" 
meta-data), or in other file(s). A movie structured into one file is commonly called 
"flat", and is self-contained. Non-flat movies can be structured to reference some, or 
all, of the media data in other files. 

As such, the format is generally suited for optimization in different 
applications. For example, when editing (compositing), data need not be rewritten as 
edits are applied and media is re-ordered; the meta-data file may be extended and 
temporal mapping information adjusted. When edits are complete, the relevant media 
data and meta-data may be rewritten into a single, interleaved, and optimized file for 
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local or network access. Both the structured and the optimized files are valid 
QuickTime files, and both may be inspected, played, and reworked. 

The use of structured ("non-flat") files enables the same basic media data to be 
used and re-used in any number of presentations. This same advantage applies when 

5 serving, as will be seen below. 

In both editing and serving, this also permits a number of other files to be 
treated as part of a movie without copying the media data. Thus editing and serving 
may be done directly from files such as Sun Microsystem's "au" audio format or the 
AVI video format, greatly extending the utility of these formats. 

10 The QuickTime file is divided into a set of objects, called atoms. Each object 

starts with an atom header, which declares its size and type: 

class Atom { 

int{32) size; 
char type[4]; 
15 byte contents [ ] ; 

} 

The size is in bytes, including the size and type header fields. The type field is 
four characters (usually printable), to permit easy documentation and identification. 
The data in an object after the type field may be fields, a sequence of contained 
20 objects, or both. 
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A file therefore is simply a sequence of objects: 

class File { 
Atom [ ] ; 

} 

5 The two important top-level objects are the media-data (mdat) and the meta- 

data (moov). 

The media-data object(s) contain the actual media (for example, sequences of 
sound samples). Their format is not constrained by the file format; they are not 
usually objects. Their format is described in the meta-data, not by any declarations 
10 physically contiguous with them. So, for example, in a movie consisting solely of 
motion-JPEG, JPEG frames are stored contiguously in the media data with no 
intervening extra headers. The media data within the media data objects is logically 
divided into chunks; however, there are no explicit chunk markers within the media 
data. 

15 When the QuickTime file references media data in other files, it is not required 

that these ^secondary' files be formatted according to the QuickTime specification, 
since such media data files may be formatted as if they were the contents of a media 
object Since the QuickTime format does not necessarily require any headers or other 
information physically contiguous with the media data, it is possible for the media data 

20 to be files which contain ^foreign' headers (e.g. UNIX ".au" files, or AVI files) and 
for the QuickTime meta-data to contain the appropriate declarative information and 
reference the media data in the 'foreign' file. In this way the QuickTime file format 
can be used to update, without copying, existing bodies of material in disparate 
formats. The QuickTime file format is both an estabUshed format and is able to work 

25 with, include, and thereby bring forward, other established formats. 



r 
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Free space (e.g. deleted by an editing operation) can also be described by an 
object. Software reading a file that includes free space objects should ignore such free 
space objects, as well as objects at any level which it does not understand. This 
permits extension of the file at virtually any level by introducing new objects. 
5 The primary meta-data is the movie object. A QuickTime file has exactly one 

movie object which is typically at the beginning or end of the file, to permit its easy 
location: 

class Movie { 

int(32) size; 
10 char type [4] = 'moov'; 

MovieHeader inh; 
contents Atom[]; 

} 

The movie header provides basic information about the overall presentation (its 
15 creation date, overall timescale, and so on). In the sequence of contained objects there 
is typically at least one track, which describes temporally presented data. 

class Track { 

int(32) size; 
char type[4] = 'trak'; 

20 TrackHeader th; 

contents Atom[] ; 

} 

The track header provides relatively basic information about the track (its ID, 
timescale, and so on). Objects contained in the track might be references to other 
25 tracks (e.g. for complex compositing), or edit lists. In this sequence of contained 
objects there may be a media object, which describes the media which is presented 
when the track is played. 



r 
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The media object contains declarations relating to the presentation required by 
the track (e.g. that it is sampled audio, or MIDI, or orientation information for a 

SDscene). The type of track is declared by its handler: 

class handler { 
5 int{32) size; 

char type [4] = 'hdlr'; 

int{8) version; 
bit (24) flags; 

char handlertype[4] ; — whir for media handlers 

10 char handlersubtype[4] — vide for video, soun for 

audio 

char manufacturer [4] ; 

bit ( 32 ) handlerf lags ; 

bit (32) handlerf lagsmask; 

15 string componentname; 

} 

Within the media information there is likewise a handler declaration for the 
data handler (which fetches media data), and a data information declaration, which 
defines which files contain the media data for the associated track. By using this 
20 declaration, movies may be buih which span several files. 

At the lowest level, a sample table is used which relates the temporal aspect of 

the track to the data stored in the file: 

class sampletable { 

int(32) size; 
25 char type [4] = 'stbl'; 

sampledescription sd; 

timetosample tts; 

syncsainpletable syncs ; 

sampletochunk stoc ; 
30 samplesize ssize; 

chunkof f set cof f set ; 

shadowsync ssync; 

} 

The sample description contains infomiation about the media (e.g. the 
35 compression formats used in video). The time-to-sample table relates time in die 
track, to the sample (by index) which should be displayed at that time. The sync 
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sample table declares which of these are sync (key) samples, not dependent on other 
samples. 

The sample-to-chunk object declares how to find the media data for a given 

sample, and its description given its index: 

5 class sampletochunk { 

int(32) size; 
char type [4] = ' stsc ' ; 

int(8) version; 
bits (24) flags; 
10 int{32) entrycoiint; 

for (int i=0; i<entrycount ; i++) { 

int (32) f irstchiink; 

int (32) samplesperchunk; 

int (32) sampledescriptionindex; 

15 } 

} 

The sample size table indicates the size of each sample. The chunkoffset table 
indicates the offset into the containing file of the start of each chunk. 

Walking the above-described structure to find the appropriate data to display 
20 for a given time is fairly straightforward, generally involving indexing and adding. 
Using the sync table, it is also possible to back-up to the preceding sync sample, and 
roll forward 'silently* accumulating deltas to a desired starting point. 

Figure 1 shows the structure of a simple movie with one track. A similar 
diagram may be found in the QuickTime file format documentation, along with a 
25 detailed description of the fields of the various objects. QuickTime atoms (objects) are 
shown here with their type in a grey box, and a descriptive name above. This movie 
contains a single video track. The frames of video are in the same file, in a single 
chunk of data. It should be noted that the *chunk' is a logical construct only; it is not 
an object. Inside the chunk are frames of video, typically stored in their native form. 
30 There are no required headers or fields in the video frames themselves. 
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Figure 2 is a diagram of a self-contained file with both an audio and a video 
track. Fewer of the atoms are shown here, for brevity; the pointers from the tracks 
into the media data are, of course, the usual sample table declarations, which include 
Xkmng information. 

5 The QuickTime file format has a number of advantages, including: 

1) Scalability for size and bit-rates. The meta data is flexible, yet compact. This 
makes it suitable for small downloaded movies (e.g. on the Intemet) as well as 
providing the basis for a number of high-end editing systems. 

10 

2) Physical structure is independent of the logical and temporal structure. This 
makes it possible to optimize the physical structure differently depending on the 
use the file will have. In particular, it means that a single file format is suitable 
for authoring and editing; downloading or placing on CDROMs; and for 

15 streaming. 

3) The file format has proven capable of handling a very broad variety of codec 
types and track types, including many not known at the time the format was 
designed. This proven ability to evolve in an upwards-compatible fashion is 

20 fundamental to the success of a storage format. 



Scalable, or layered, codecs can be handled in a number of ways in the 
QuickTime file format. For a streaming protocol which supports scalability, the 
samples may be tagged with the layer or bandwidth threshold to be met for 

25 transmitting the samples. 

Tracks which form a set of alternatives (e.g. different natural language sound 
tracks) can be tagged so that only one is selected for playback. The same structure can 
be used to select alternatives for streaming (e.g. for language selection). This 
capability is described in fiarther detail in the QuickTime file format. 

30 When QuickTime displays a movie or track, the appropriate media handler 

accesses the media data for a particular time. The media handler must correcdy 
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interpret the data stream to retrieve the requested data. For example, with respect to 
video media, the media handler typically traverses several atoms to find the location 
and size of a sample for a given media time. The media handler may perform the 
following: 

5 

1. Determine the time in the media time coordinate system. 

2. Examine the time-to-sample atom to determine the sample number that contains 
the data for the specified time. 

10 

3. Scan the sample-to-chunk atom to discover which chunk contains the sample in 
question. 

4. Extract the offset to the chunk firom the chunk offset atom. 

15 

5. Find the offset within the chunk and the sample's size by using the sample size 
atom. 

It is often desirable to transmit a QuickTime file or other types of time related 
sequences of media data over a data communication medium, which may be associated 

20 with a computer network (e.g. the Internet). In many computer networks, the data 
which is transmitted into the network should generally be in a packet form. Normally, 
time related sequences of media data are not in the proper packetized format for 
transmission over a network. For example, media data files in the QuickTime format 
are not in a packetized format. Thus, there exists a need to collect the data, sometimes 

25 referred to as streaming data, into packets for transmission over a network. 

One prior approach to address the problem of transmitting time related 
sequences of media data over a network is to send the media file over the network 
using a network or transmission protocol, such as the Hypertext Transfer Protocol 
(HTTP). Thus, the media file itself is sent from one computer system over the 

30 network to another computer system. However, there may be no desire to retain the 
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media file at the receiving computing system. That is, when the media file is received 
and viewed or listened to at the receiving computer system, there may be no desire by 
the user of that receiving computer system to store a copy of the file, for example, if 
the receiving computing system is a network computer or a computer with low storage 
5 capacity. 

Another alternative approach to solving the problem of how to collect data for 
transmission by packets over a network is to prepare a file which contains the network 
protocol data units in the file for a particular transmission protocol. In a sense, such a 
file may be considered a packetized file which is stored in essentially the same format 

10 as it will be transmitted according to the particular transmission protocol. Performing 
this operation generally involves storing the file in a packetized form for a particular 
network protocol at a particular data transmission rate and a particular media file 
format. Thus, for each different transmission protocol at a particular data 
transmission rate, the file will essentially be replicated in its packetized form. The 

15 fixed form of such files may restrict their applicability/compatibility and make it 
difficult to view such files locally. Thus, such an approach may greatly increase 
storage requirements in attempting to provide the file in various transmission protocols 
at various different data transmission rates. Moreover, each packetized file generated 
according to this alternative prior approach is generally limited to a particular media 

20 file format, and thus, other media file formats for the same media object (e.g. a digital 
movie) are typically packetized and stored on the sending computer system. 

Yet another approach to solving the problem of how to stream time related 
sequences of media data is to perform the packetization of the media data when 
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required on the transmitting system according to the particular transmission protocol 
which is desired. This processing requires, in many cases, a relatively considerable 
amount of time, and thus, may slow the performance of the transmitting system. 

Thus, it is desirable to provide an unproved method and apparatus for 
transmitting time related sequences of media data. 



-13- 

SUMMARY OF THE INVENTION 

The present invention provides methods and apparatuses for processing media 
data for transmission in a data communication medium. In one embodiment, a set of 

5 data indicates how to transmit a time related sequence of media data according to a 
transmission protocol. The set of data, according to one embodiment, includes a time 
related sequence of data which is associated with the time related sequence of media 
data. According to one aspect of the invention, the set of data may be utilized by a 
digital processing system to transmit the time related sequence of media data (e.g., by 

10 packets generated according to the transmission protocol and the set of data). 



BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows an example of the structure of a simple movie with one track 
in the prior art. 

Figure 2 is an example of a self-contained movie file of the prior art. 
5 Figure 3 is a flowchart showing one example of a method according to the 

present invention. 

Figure 4 shows an example of a hint track of the present invention. 
Figure 5 shows another example of a hint track of the present invention. 
Figure 6 is a diagram of a network of computer systems in which media data 
10 may be exchanged and/or processed, according to one embodiment of the present 
invention. 

Figure 7 is a block diagram of a digital processing system which may be used 
in accordance with one embodiment of the present invention. 

Figure 8 is a block diagram of a system that utilizes hints to transfer media 
15 data, according to one embodiment of the invention. 

Figure 9 is a block diagram of a system that utilizes hints to transfer media 
data, according to one embodiment of the invention. 

Figure 10 is a flow diagram illustrating a method for generating hints for 
providing media data transmission, according to one embodiment of the invention. 
20 Figure 1 1 is a flow diagram illustrating a method of processmg media data 

received by a receiving system in accordance with hints, according to one embodiment 
of the invention. 
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Figure 12 is an example of a machine readable storage medium that may be 
accessed by a digital processing system, such as a generator, according to one 
embodiment of the invention. 

Figure 13 is an example of a machine readable storage medium that may be 
5 accessed by a digital processing system, such as a server, according to one 
embodiment of the invention. 

Figure 14 is an example of a machine readable storage medium that may be 
accessed by a digital processing system, such as a receiving system or other digital 
processing system, according to one embodiment of the invention. 
10 Figure 15 is a diagram of a data storage and/or communication medium having 

stored/transported thereon media and hint information, according to one embodiment 
of the invention. 
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DETAILED DESCRIPTION 
The present invention provides methods and apparatuses for allowing the 
transmission, and particularly the packetized transmission of time related sequences of 
media data, which may include, for example, video, audio, video and audio, etc., 

5 over a communication media, such as in a computer network. 

In one embodiment of the present invention, a digital processing system 
creates a set of data for indicating how to transmit a time related sequence of media 
data according to a transmission protocol. Typically, this set of data is stored on a 
storage device coupled to the digital processing system. Further, this set of data is a 

10 time related sequence of data associated with the time related sequence of media data. 

The present invention may be implemented entirely in executable computer 
program instructions which are stored on a computer readable media or may be 
implemented in a combination of software and hardware, or in certain embodiments, 
entirely in hardware. Typically, a server computer system coupled to a network will 

15 create the set of data, which may be referred to as a hint track and will store this hint 
track in a storage device which is coupled to the server computer system. When a 
client computer system requests a presentation (e.g. a viewing or listening or viewing 
and listening) of a media data file, the server system uses the hint track to determine 
how to packetize the media data for transmission to the client computer system. It will 

20 be appreciated that the present invention is generally applicable to time related 

sequences of media data, and that QuickTime is represented herein as one example of 
this general applicability. Thus, the invention should not necessarily be limited to 
QuickTime. 
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Figure 3 shows one example of a method according to the present invention. 
The method 300 shown in Figure 3 begins in step 301, in which the media file format 
for the particular media data which is desired to be transmitted is determined. In step 
303, the particular transmission protocol or protocols which are desired to be used is 
5 also determined. However, steps 301 and 303 are optional, for example, in the case 
where the same media file format is always transmitted using the same transndission 
protocol. 

In step 305, a digital processing system, such as a server computer system, 
creates and stores the hints for packetizing a time related sequence of media data in a 

10 media file. Alternatively, one computer system may create the hints and provide them 
to another system, such as a server computer system, which stores them for later use 
in a transmission process. The packetization allows the transmission over a network 
or communication media according to the desired transmission protocol which was 
determined in step 303. In one embodiment of the present invention, the hints are 

1 5 stored as a track of time related sequence of hints which refers to, but which in one 
embodiment, is separate from other tracks of media data. The track of hints, in one 
embodiment of the present invention, may be stored separately from the media data to 
which it refers. As such, the track of hints may be stored in a file which is distinct 
from another file containing the media data which is referred to by the track of hints, 

20 or the track of hints may be stored in a hint area in the file containing the media data 
which is separate and distinct from the data area containing the actual media data. In 
one embodiment of the invention, a hint track, or portion thereof, may be interpreted 
as executable instructions by the server, which executable instructions cause the server 
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to packetize a time related sequence of data, which is typically, but not necessarily, 
time-based media data. In one embodiment of the present invention, the hints are 
stored on the storage device which is coupled to the transmitting digital processing 
system. 

5 In step 307, the data which is packetized according to the hints, is transmitted 

from a transmitting system, such as a server computer system, to a receiving system. 
This media data is transmitted by packetizing the media data according to the hints. In 
one alternative embodiment of the invention, the server computer system may decide 
not to use the hints and to send the media data by an altemative packetization process. 

10 In step 309, the receiving system presents the media object which is 

represented by the media data. Typically, this presentation (which may be a viewing 
and listening of a media object or merely a viewing or merely a listening of the media 
object) is performed as the packetized data is received at the receiving system. The 
packetized data may, in one embodiment of the present invention, but need not be, 

15 stored on the receiving system. Thus the presentation of the data is ephemeral in the 
sense that once the presentation is over, there is no local copy at the receiving system. 
In another embodiment, presentation of the media object may take place on the server 
system subsequent to creating hints for the media data representing the media object. 
In one embodiment of the invention, the media data is not necessarily (re)formatted, 

20 copied, etc., for packetization according to hints. 

In step 3 1 1, the receiving system may optionally reassemble the media file if 
the media file as received has been stored on the receiving system. It will be 
appreciated that the various steps of the method shown in Figure 3 may be performed 
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in a different order than the one shown and described above and/or some of the steps 
may be performed simultaneously. For example, in one embodiment, steps 309 and 
3 1 1 are performed in parallel. 

A particular implementation with QuickTime according to one embodiment of 
5 the present invention will now be described. Li one embodiment of the present 

invention, a presentation which can be both viewed locally to the file (e.g., at a server, 
generator, etc.), and streamed over a network within a QuickTime movie is provided. 
In general, the streaming server (or another system) should have information about the 
data units to stream, then: composition and timing. Since such information is typically 

10 temporal it may be described in tracks. A server may perform packetization and 

determine protocol information, for example, by using the same indexing operations 
as would be used to view a presentation. 

The tracks which contain instructions for the servers are sometimes referred to 
as 'hint' tracks, since such tracks represent a set of data to direct the server in the 

15 process of forming and transmitting packets. The QuickTime file format supports 
streaming of media data over a network as well as local playback. The process of 
sending protocol data units is time-based, just like the display of time-based data, and 
is therefore suitably described by a time-based format A QuickTime file or *movie' 
which supports streaming includes information about the data units to stream. This 

20 infomaation is included in additional tracks of the file called "hint" tracks. 

Hint tracks contain instructions for a streaming server (or other digital 
processing system) which assist in the formation of packets. These instructions may 
contain immediate data for the server to send (e.g. header information) or reference 
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segments of the media data. In one embodiment of the present invention, instructions 
are encoded in the QuickTime file in the same way that editing or presentation 
information is encoded in a QuickTime file for local playback. Instead of editing or 
presentation information, information may be provided which may allow a server to 
5 packetize the media data in a manner suitable for streanMig using a specific network 
transport. 

In one embodiment of the present invention, the same media data is used in a 
QuickTime file which contains hints, whether it is for local playback, or streaming 
over a number of different transport types. Separate *hint' tracks for different 

10 transport types may be included within the same file and the media may play over all 
such transport types without making any additional copies of the media itself. In 
addition, existing media may be made streamable by the addition of appropriate hint 
tracks for specific transports. According to one aspect of the invention, media data 
itself need not be recast or reformatted. 

15 Therefore the samples in a hint track generally contain instructions to form 

packets. These instructions may contain immediate data for the server to send (e.g. 
header information) or reference segments of the media data in another track. 

In one embodiment of the present invention, a three-level design is utilized 
such that: 



20 



1) 



The media data is represented as a set of network-independent tracks, 
which may be played, edited, and so on, as normal; 



25 



2) 



There is a common declaration and base structure for server hint tracks; 
this common format is protocol independent, but contains the declarations 
of which protocol(s) are described in the server track(s); 
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3) There is a specific design of the server hint tracks for each protocol which 
may be transmitted; all these designs use the same basic structure. For 
example, there may be designs for RTP (for the Intemet) and MPEG-2 
transport (for broadcast), or for new standard or vendor-specific 
5 protocols. 

In one embodiment of the present invention, the resulting streams, sent by the 
servers under the direction of the hint tracks, are normal streams, and do not 
necessarily include a trace of QuickTime information. This embodiment of the 

10 invention does not require that QuickTime, or its structures or declaration style, 

necessarily be either in the data on the transmission medium (e.g. network cable) or in 
the decoding station. For example, a file using H.261 video and DVI audio, streamed 
under RTP, may result, in one embodiment of the present invention, in a packet 
stream which is fully compliant with the IETF specifications for packmg those 

15 codings into RTP. 

In one embodiment of the invention, hint tracks are built and flagged so that 
when the presentation is viewed locally, the hint tracks are essentially ignored by a 
receiving system. 

In one embodiment, a time related sequence of media data, which may, for 
20 example, include video, audio, etc., may be packetized by a digital processing system, 
and then presented on the same digital processing system. Furthermore, packetization 
may be ephemeral, such that the time related sequence being presented, stored, read, 
etc., is also packetized "on the fly." In one embodiment, hints may refer to media data 
that has not been copied, formatted, etc.; for example, the media data to which hints 
25 refer may be stored in original format on a read-only memory, etc. 
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In one embodiment, the Scime hinting routine that provides packetization also 
presents the media as packetization is performed. In altemative embodiments of the 
invention, a packetized file of time related media data may be generated according to 
hint tracks and stored, for example, for later transmission, 
5 Figure 4 illustrates utilization of hint tracks for transporting media data, 

according to one embodiment of the invention. In Figure 4, a hint track 401 is shown 
for the media track 403. Each hint track sample, such as hint track sample 405 — 
which describes how to form an RTP packet — ^may contain a header, and may 
reference some data from an associated media track — ^in this case, a video track 403. 

10 In the embodiment shown in Figure 4, the media data (the video frames) and the RTP 
hints have been interleaved so that the associated media file may be read relatively 
easily. In this example, each frame is shown as fitting into a single RTP packet. Of 
course, it is possible to split frames into several packets when needed. Conversely, 
multiple frames can, if desired, be placed in a single packet, which is conmionly 

1 5 performed with audio data. 

As discussed above, the logical structure described above need not imply 
physical structure. The meta data may be cached in memory, and the hint track 
samples physically interleaved with the media samples to which they refer (as is 
shown in Figure 4). 

20 Altematively, it is possible to write a new set of meta data and media data, 

containing the hint tracks, which references and augments the meta data and media 
data in an existing presentation. Figure 5 illustrates utilization of hint tracks to 
reference media data in a separate file, according to one embodiment of the invention. 



Y 
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In Figure 5, two movie files 502 and 504 are shown, each with their own meta-data. 
The first, the movie file 502, includes a video track. The second, the movie file 504, 
contains both a video track and a hint track, but the meta-data declares that the media 
data for the video track is in the first movie 502. Thus the hints associated with the 
5 movie file 504 also point to the media data in the first movie 502. 

In one embodinient of the present invention, a media file may contain 
packetization hint tracks for multiple protocols. As such, each track may contain 
declarations of the protocol (and protocol parameters, if appropriate) for which the 
hint track is appropriate. These tracks may all, of course, reference media data from 

10 the basic media tracks in the file. The desire for protocol independence and 
extensibility may be met in the described manner. 

In one embodiment of the present invention, hint tracks need not use all the 
data in the media tracks. The hint tracks may use a subset of the data (e.g. by omitting 
some video frames) to reach a bandwidth threshold, or for other reasons. Since 

15 multiple hint tracks may be provided for the same protocol, differing subsets of the 
same basic media information at different rates may be provided. As such, the present 
invention may provide improved scalability over prior methods and apparatuses. 

It should be emphasized that though the hint tracks themselves, and the 
QuickTime meta-data, should, in one embodiment, be in QuickThne files, the base 

20 media can be left in any file type which QuickTime can import and reference in place. 
In one embodiment of the present invention, the meta-data in the movie file may 
include a data reference which declares that the media data is in another file. The 
sample table offsets and pointers may thus refer to data in this 'foreign' file. Thus, 
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according to one embodiment of the present invention, existing legacy formats such as 
"au" audio files, "AVT' audio/video files, and MIDI files, may be streamed without 
requiring the copying or reformatting of the base media data. Since the base media 
data is not written to, but merely augmented by QuickTime declarations and hint 
5 information in separate files, the base media data may also be provided on read-only 
machine readable media such as CDROM. 

In one embodiment of the present invention, the hint tracks embody the results 
of off-line computation and are typically optimized to provide the server with 
information to support packetization, and if needed, multiplexing. 
10 Example hints, for example, for RTP (the IETF standard real-time protocol) 

and MPEG-2 transport are shown in Appendixes A-C. 

In one embodiment of the present invention, a single file may support hint 
tracks for multiple protocols, or multiple different parameterizations of the same 
protocols, without undue space overhead. New protocols, and their associated hint 
15 tracks, may be designed without disrupting systems relying on existing protocols. 
Thus the invention, at least in one embodiment, is protocol-neutral. 

In the QuickTime file format, a track may be added to the movie by updating 
or copying and augmenting the meta-data. If the media data is in files separate from 
the meta-data, or optimized interleave is not required, this can be a relatively simple 
20 and efficient operation* 

In one embodiment of the present invention, tracks may be extracted by 
building a new set of movie meta-data which contains only one track, and which can, 
if desired, reference the media data in the original. 
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For example, in one embodiment of the present invention, a new audio track 
may be added which is marked as being an alternative to a set of other audio tracks. If 
it is also marked with the language code (e.g. French, or Tagalog), then the 
appropriate track may be selected at presentation time. 
5 SMPTE time-code tracks are an example of elementary streams which may be 

present, added, or removed, as need arises, according to one embodiment of the 
invention. 

According to one aspect of the invention, hint tracks may permit the 
development of new formats for new protocols without causing compatibility issues 
10 for existing servers or local playback. In addition, new media tracks may be added 
over the life of the file format while maintaining backwards compatibility. 

In one embodiment of the present invention, the areas of extensibility include: 

a) New track types which can be defined for media types not covered by the current 
15 QuickTime file format (e.g. laboratory instrument readings). 

b) New coding types for existing tracks which may be defined (e.g. video or audio 
codecs). There is explicit provision for their codec-specific initialization 
information. 

20 

c) New hint track types which may be defined for new protocols, and a file which 
may contain hint information for more than one protocol without incurring a 
space overhead for the media data itself. 

Existing content on read-only media may be used with the present invention 
25 (e.g., prepackaged movies on CD ROM, DVD, etc.). 

Furthermore, according to one aspect of the invention, various "foreign" file 
formats may be used. In one embodiment of the present invention, for example, if the 
existing content is either in QuickTime format, or can be imported, it may be edited 
and streamed without requiring copying or re-formatting. 
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In one embodiment of the present invention, if a codec supports striping of the 
media data to achieve scalability of bandwidths, then these striped bandwidths may be 
represented using multiple stream tracks. Each track may represent a different 
bandwidth. Tracks may be grouped together in selected subsets of the basic media. 
5 In one embodiment of the present invention, if a protocol supports bandwidth 

scalability, then the hint track itself may contain information for each protocol data unit 
(sample in the hint track). Information may include the bandwidth threshold above 
which the protocol data unit should be delivered to die network. Thus, hint tracks 
may indicate an available bandwidth as being high, low, etc., and/or other information 

10 relating to bandwidth for data transmission. 

In one embodiment of the present invention, if the protocol is a multiplexing 
protocol (e.g. MPEG-2 transport) then different hint tracks may be built which use a 
different subset of the elementary stream tracks to achieve different data-rates. Hence, 
some tracks may be omitted entirely for low bit-rate transmission. 

15 In one embodiment of the present invention, if it is desired to record the base 

data using different codecs, then those tracks may be formed into a group of 
alternatives, and only one selected for presentation. The selection of which track to 
use for presentation is typically protocol-dependent and may be achieved by using the 
hint track approaches described herein. 

20 In one embodiment of the present invention, encryption may also be pre- 

applied to a media file. In this case, the encrypted data may be stored in either (a) a 
new elementary stream (a new track) which is linked to the original media data (or the 
original media data may be removed if it is no longer needed) or (b) the hint track 
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itself. In case (b), it is possible that the hint track does not extract any data from the 
elementary unencrypted stream on the fly. Thus, all of the media data may be in the 
hint track as well as the streaming packet protocol data unit information, because the 
media data may be transformed by encryption. 
5 As an example of embedded object content information, the IETF session 

description information for a whole movie, and for individual tracks, may be stored in 
the meta-data for the RTP hint tracks, as user atoms. 

In one embodiment of the present invention, a file format typically contains 
both media data in a playable format, and streaming information. In one embodiment, 
10 it is possible to stream directly from this format with relatively low overhead, while 
preserving the media independence, protocol independence, and ability to present the 
media locally. 

According to one aspect of the invention, hint tracks may abstract detailed 
knowledge of codecs, timing and packetization, into an off-line preparation process. 

15 Thus, following the hint tracks to generate the data stream may be relatively simple 
and require no specialized knowledge of the media being streamed. Thus, decoupling 
of a server, for example, from the details of the data content may be provided, 
according to one aspect of the invention. 

In one embodiment of the present invention, a set of hint tracks may be used to 

20 construct a file which is directly optimized for streaming — ^for example, by laymg out 
network PDUs on disk at logical disk boundaries, in the time sequence in which they 
should sent. Such a file may no longer be a general presentation, but may be 
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streamed. In one embodiment, packetized files created with hint tracks may be stored 
and, for example, later optimized for streaming. 

In one embodiment of the present invention, by encapsulating foreign file 
formats, media data may be retained in other formats while still be published in 
5 QuickTime. For example, an existing format may be directly encapsulated into a new 
media data file by applying the proper wrapper, or may be left intact and refeired to in 
segments or as a whole by the hint track, allowing the legacy formats to be streamed 
without copying. A single movie may contain pieces selected from multiple legacy 
formats. This invention does not constrain the base media format. 

10 In general, a common format which spans capture, authoring and editing, 

download and streaming, will generally provide flexibility. Material may be reworked 
after use, or used in multiple ways, without being copied or re-formatted. In one 
embodiment of the present invention, it is possible to re-work and re-use material 
which has been hinted, by stripping the hint tracks, using standard editors, and then 

1 5 re-hinting after editing is completed. 

If it is desired that a media file be downloaded for local viewing, an optimized 
interleaved file may be built for that purpose, with the streaming meta-data in a 
separate declaration file referencing the same base media data. The download may 
not, therefore, include the streaming information, and yet the media data may be 

20 present only once at a streaming server. 

By separating logical structure from physical structure, the physical structure 
of the file may be optimized differently depending on the application (e.g. editing, 
local viewing, streaming). 
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By permitting the existence of multiple hint tracks for each media track, in one 
embodiment of the present invention, the file may be published by streaming over 
multiple protocols, without requiring multiple copies of the media. 

Figure 6 is a diagram of a network of computer systems in which media data 
5 may be processed, according to one embodiment of the present invention. As shown 
in Figure 6, a number of client computer systems, one or more of which may 
represent one implementation of the receiving system described above with reference 
to Figure 3, are coupled together through an Intemet 622, It will be appreciated that 
the term "Intemet" refers to a network of networks. Such networks may use a variety 

10 of protocols for exchange of information, such as TCP/IP, ATM, SNA, SDI, etc. 
The physical connections of the Internet and the protocols and communication 
procedures of the Intemet are well known to those in the art. Access to the Intemet 
103 is typically provided by Intemet service providers (ISPs), such as the ISP 624 
and the ISP 626. Users on client systems, such as the client computer systems 602, 

15 604, 618, and 620, generally obtain access to the Intemet through Intemet service 
providers, such as ISPs 624 and 626. Access to the Intemet may facilitate transfer of 
information (e.g., email, text files, media files, etc.) between two or more digital 
processing systems, such as the client computer systems 602, 604, 618, and 620 
and/or a Web server system 628. For example, one or more of the client computer 

20 systems 602, 604, 618, and 620 and/or the Web server 628 may provide media data 
(e.g., video and audio, or video, or audio) to another one or more of the client 
computer systems 602, 604, 618, and 620 and/or the Web server 628. Such may be 
provided in response to a request. As described herein, such media data may be 
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transferred in the system 600 according hints. Such hints, in one embodiment of the 
invention, may be created according to a specific format of the media data and/or a 
specific data communication (e.g., network) protocol(s). 

The Web server 628 is typically comprised of at least one computer system to 

5 operate with one or more data communication protocols, such as the protocols of the 
World Wide Web, and as such, is typically coupled to the Internet 622. Optionally, 
the Web server 628 may be part of an ISP which may provide access to the Internet 
and/or other network for client computer systems. The client computer systems 602, 
604, 618, and 620 may each, with appropriate web browsing software, access data, 

10 such as HTML documents (e.g., Web pages), which may be provided by the Web 
server 628. Such data may provide media, such as QuickTime movies, which may be 
presented by the client computer systems 602, 604, 618, and 620. 

The ISP 624 provides Internet connectivity to the client computer system 602 
via a modem interface 606, which may be considered as part of the client computer 

15 system 602. The client computer system may be a conventional computer system, 
such as a Macintosh computer, a "network" computer, a handheld/portable computer, 
a Web TV system, or other types of digital processing systems (e.g., a cellular 
telephone having digital processing capabilities). Similarly, the ISP 626 provides 
Internet connectivity for the client computer systems 604, 618 and 620, although as 

20 depicted in Figure 6, such connectivity may vary between various client computer 
systems, such as the client computer systems 602, 604, 618, and 620. For example, 
as shown in Figure 6, the client computer system 604 is coupled to the ISP 626 
through a modem interface 608, while the client computer systems 618 and 620 are 
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part of a Local Area Network (LAN). The interfaces 606 and 608, shown as modems 
606 and 608, respectively, in Figure 6, may be an analog modem, an ISDN modem, a 
cable modem, a satellite transmission interface (e.g., "Direct PC")» a wireless 
interface, or other interface for coupling a digital processing system, such as a client 

5 computer system, to another digital processing system. The client computer systems 
618 and 620 are coupled to a LAN bus 612 through network interfaces 614 and 616, 
respectively. The network interfaces 614 and 616 may be an Ethernet-type, 
Asynchronous Transfer Mode (ATM), or other type of network interface. The LAN 
bus is also coupled to a gateway digital processing system 610, which may provide 

10 firewall and other Internet-related services for a LAN. The gateway digital processing 
system 610, in turn, is coupled to the ISP 626 to provide Internet connectivity to the 
client computer systems 618 and 620. The gateway digital processing system 610 
may, for example, include a conventional server computer system. Similarly, the 
Web server 628 may, for example, include a conventional server computer system. 

15 The system 600 may allow one or more of the client computer systems 602, 

604, 618, and 620 and/or the Web server 628 to provide media data (e.g., video and 
audio, or video, or audio) to another one or more of the client computer systems 602, 
604, 618, and 620 and/or the Web server 628. Such data may be provided, for 
example, in response to a request by a receiving system, which may be, for example, 

20 one or more of the client computer systems 602, 604, 618, and 620. As described 
herein, such media data may be transferred in the system 600 according hints or hint 
tracks. Such hints, in one embodiment of the invention, may be created according to a 
specific format of the media data and/or a specific data communication (e.g., network) 



protocol(s) to allow, according to one aspect of the invention, packetization of media 
data. 

Figure 7 is a block diagram of a digital processing system which may be used 
in accordance with one embodiment of the present invention. For example, the digital 

5 processing system 650 shown in Figure 7 may be used as a client computer system, a 
Web server system, a conventional server system, etc. Furthermore, the digital 
processing system 650 may be used to perform one or more functions of an Internet 
service provider, such as the ISP 624 or 626. The digital processing system 650 may 
be interfaced to external systems through a modem or network interface 668. It will 

10 be appreciated that the modem or network interface 668 may be considered as part of 
the digital processing system 650. The modem or network interface 668 may be an 
analog modem, an ISDN modem, a cable modem, a token ring interface, a satellite 
transmission interface, a wireless interface, or other interface(s) for providing a data 
conoimunication link between two or more digital processing systems. 

15 The digital processing system 650 includes a processor 652, which may 

represent one or more processors and may include one or more conventional types of 
such processors, such as a Motorola PowerPC processor, an Intel Pentium (or x86) 
processor, etc. A memory 155 is coupled to the processor 652 by a bus 656. The 
memory 155 may be a dynamic random access memory (DRAM) and/or may include 

20 static RAM (SRAM). The processor may also be coupled to other types of storage 

areas/memories (e.g., cache. Flash memory, disk, etc.), which could be considered as 
part of the memory 155 or separate from the memory 155. 
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The bus 656 further couples the processor 652 to a display controller 658, a 
mass memory 662, the modem or network interface 668, and an input/output (I/O) 
controller 664. The mass memory 662 may represent a magnetic, optical, magneto- 
optical, tape, and/or other type of machine-readable medium/device for storing 

5 information. For example, the mass memory 662 may represent a hard disk, a read- 
only or writeable optical CD, etc. The display controller 658 controls in a 
conventional manner a display 660, which may represent a cathode ray tube (CRT) 
display, a liquid crystal display (LCD), a plasma display, or other type of display 
device. The I/O controller 664 controls I/O device(s) 666, which may include one or 

10 more keyboards, mouse/trackball or other pointing devices, magnetic and/or optical 
disk drives, printers, scaimers, digital cameras, microphones, etc. 

It will be appreciated that the digital processing system 650 represents only 
one example of a system, which may have many different configurations and 
architectures, and which may be employed with the present invention. For example, 

15 Macintosh and Intel systems often have multiple busses, such as a peripheral bus, a 
dedicated cache bus, etc. On the other hand, a network computer, which may be used 
as a digital processing device of the present invention, may not include, for example, a 
hard disk or other mass storage device, but may receive routines and/or data from a 
network connection, such as the modem or interface 668, to be processed by the 

20 processor 652. Similarly, a Web TV system, which is known in the art, may be 
considered to be a digital processing system of the present invention, but such a 
system may not include one or more I/O devices, such as those described above with 
reference to I/O device(s) 666. Additionally, a portable communication and data 
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processing system, which may employ a cellular telephone and/or paging capabilities, 
may be considered a digital processing system which may be used with the present 
invention. 

In the system 650 shown in Figure 7, the mass memory 662 (and/or the 
5 memory 654) may store media (e.g., video, audio, movies, etc.) which may be 
processed according the present invention (e.g., by way of hints). Alternatively, 
media data may be received by the digital processing system 650, for example, via the 
modem or network interface 668, and stored and/or presented by the display 660 
and/or I/O device(s) 666. In one embodiment, packetized media data may be 

10 transmitted across a data communication network, such as a LAN and/or the Internet, 
in accordance with hint tracks. On the other hand, the processor 652 may execute one 
or more routines to use a file with one or more hint tracks, or alternatively, to create 
one or more hint tracks, to process media (e.g., a pre-packaged movie, audio file, 
video file, etc.) for presentation or packetization according to the hint tracks. Such 

15 routines may be stored in the mass memory 662, the memory 664, and/or another 
machine-readable medium accessible by the digital processing system 650. In one 
embodiment, the digital processing system 650 may process media data having hint 
tracks embedded therein. Similarly, such embedded media data may be stored in the 
mass memory 662, the memory 664, and/or another machine-readable medium 

20 accessible by the digital processing system 650. 

Figure 8 is a block diagram of a system that utilizes hints to transfer media 
data, according to one embodiment of the invention. The system 680 shown in Figure 
8 includes a receiving system, which is depicted as a client data processing system 



682 coupled to a server 694, via a data communication link 686. The server 694 
and/or client data processing system may, for example, represent one or a combination 
of the devices/systems described with reference to Figures 6 and 7. 

The server 694 includes a hint generation and processing unit 688, a media 
5 processing unit 690, and a data communication unit 692, each of v^hich may include 
hard-wired circuitry or machine-executable instructions or a combination thereof. 
Furthermore, at least a portion of such hard-wired circuitry and/or machine-executable 
instructions may be shared between a combination of the hint generation and 
processing unit 688, the media processing unit 690, and the data communication unit 

10 692. In one embodiment, at least one storage area/memory (e.g., a machine-readable 
medium) having appropriate routines and/or data stored therein coupled to at least one 
processor is utilized, at least in part, to implement one or a combination of the hint 
generation and processing unit 688, the media processing unit 690, and the data 
conmiunication unit 692. 

15 In one embodiment, the hint generation and processing unit 688 creates and 

stores hints for packetization of media data processed by the media processing unit 
690. As described above, the hints may be generated and stored as a separate file, 
relative to media files or may be embedded with media data. If more than one media 
format is to be processed, an appropriate foraiat may be taken into consideration by 

20 the hint generation and processing unit 688 to generate the hints. Information about 
the media format may be provided by the media processing unit 690, which may also 
provide the media data (e.g., media files of video, audio, or video and audio, etc.). 
Similarly, the data communication unit 692 may provide one or more data 
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communication (e.g., network) protocols for exchange of such media data, packetized 
according to the hints, via the data communication link 686. As such, the hint 
generation and processing unit may determine, based on media format information 
provided by the media processing unit 690 and data communication protocol 

5 information provided by the data communication unit 692, appropriate hints and 
packetization of media and/or the hints for transfer to a receiving digital processing 
system, such as the cUent data processing system 682. In one embodiment, the 
streaming of the media and hints is done in accordance with the QuickTime format. 

In response to media data and hint packets received via the data communication 

10 link 686, the cUent data processing system 682 may present a media object represented 
by the media data. Such presentation may be performed ephemerally, as described 
above. In one embodiment of the invention, the media data may optionally be stored 
by the client data processing system 682 and reassembled, for example, at a later time, 
for presentation and/or transmission by the client data processing system 682. 

15 Figure 9 is a block diagram of a system that utilizes hints to transfer media 

data, according to one embodiment of the invention. In particular, Figure 9 depicts an 
embodiment of the invention wherein a separate digital processing system, referred to 
as a generator, may generate hints (or hint tracks) to provide to another system, such a 
server, that uses the hints to packetize media data for transfer to another system, such 

20 as a client computer system. A system 696 is shown in Figure 9, which includes a 
server 700 which may exchange data, via the data communication link 686, with the 
client data processing system 682. However, in the embodiment shown in Figure 9, 
the server 700 does not generate the hints. Rather, a generator 7 10, coupled to the 



server 700 by a data communication link 708, includes a hint generation unit 712 to 
generate hints that are used to packetize media data. 

In one embodiment, the operation of the system 696 is as follows: the server 
700 makes a request to the generator 7 10 to generate hints for one or more media files 

5 containing media data. For example, the media files may be stored in the server 700 
on a machine-readable medium. The request may include information to indicate the 
format of the media file and/or a data communication protocol for transmission of the 
media data and/or other data. The data communication protocol may be related to the 
data communication link 686, which may, in one embodiment of the invention, be 

10 associated with a network connection having particular physical and logical 

characteristics to facilitate exchange of media and/or other data between the server 700 
and the client data processing system 682. In response to the request, the hint 
generation unit 712 generates appropriate hints, which may be associated with a time- 
related hint track, and provides the hints to the server 700« In response to the hints 

15 received fi"om the generator 710, via the data communication link 708, the server 700, 
and in particular, a hint processing unit 702 uses the hints to packetize the media data 
for transmission to the client data processing system 682. 

In response to media data and hint packets received via the data communication 
link 686, the client data processing system 682 may present a media object represented 

20 by the media data. Such presentation may be performed ephemerally, as described 
above. In one embodiment of the invention, the media data may optionally be stored 
by the client data processing system 682 and reassembled, for example, at a later time, 
for presentation and/or transmission by the client data processing system 682. 
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Figure 10 is a flow diagram illustrating a method for generating hints for 
providing media data transmission, according to one embodiment of the invention. In 
step 720, a media format is determined for media data to be transmitted, if more than 
one format will be used. If only one format is used, 720 may not be performed. In 
5 step 722, an appropriate data communication protocol(s) is determined, again, 

assuming that more than one (protocol) may be used. In step 724, based on the media 
format and the data communication protocol(s) (one or both of which may have been 
selected/configured), hints (e.g., hint tracks) related to media data transmission are 
created and stored. 

10 In step 726, which is optional, the hints may be transmitted to another digital 

processing system. In one embodiment of the invention, for example, the method of 
Figure 10, at least in part, may be performed exclusively by one digital processing 
system (e.g., a server). In an alternative embodiment, the method of Figure 10, at 
least in part, may be performed by two or more digital processing systems. For 

1 5 example, attributes of media data may be provided by a server or other system to 
another digital processing system, such as a generator. In response, the generator 
may determine, based on the attributes, an appropriate media format, data 
communication protocol(s), and hints for packetization of media data, which may be 
stored at the server. Altematively, the server may provide the appropriate media 

20 format and protocol(s) to the generator, which could then generate hints. The 
generator may transmit the hints to the server or other digital processing system, 
which could packetize media data according to the hints. 
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Figure 1 1 is a flow diagram illustrating a method of processing media data 
received by a receiving system in accordance with hints, according to one embodiment 
of the invention. In step 730, media data transmitted according to a receiving system 
in accordance with hints or hint tracks is received by the receiving system. In one 

5 embodiment, the receiving system may receive packetized media data, as well as 
packetized hint tracks. The hint tracks, in one embodiment of the invention, may be 
associated with at least portions of the media data. Such data may be received by the 
receiving system in response to a request that may be made by the receiving system. 
For example, in one embodiment, the receiving system may be a client computer 

10 system and the request may be made to a server or other digital processing system for 
the media data. In response, the server may generate (or have generated for it by a 
separate digital processing system) hints for packetizmg the media data, and transmit 
the packetized media data, which may include hints, to the receiving system. 

In step 732, a media object represented by the media data received by the 

15 receiving system is presented by the receiving system. For example, the media data 
may include video, audio, or combination thereof that is "presented" by the receiving 
system, for example, on a display and speaker(s). As mentioned above, the media 
data may be associated with a QuickTime movie. 

Optionally, in step 734, the media data, which may include hints, may be 

20 stored by the receiving system as a media file(s). Thus, in alternative embodiments of 
the invention, step 732 may not be performed as the media data is received, or may be 
performed before, after, or in parallel with step 734. 
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In step 734, the stored media file may optionally be reassembled and/or 
presented. As such, step 732 may be performed subsequent to step 734, 

Figure 12 is an example of a machine readable storage medium that may be 
accessed by a digital processing system, such as a generator, according to one 

5 embodiment of the invention. It will be appreciated that the actual memory that stores 
the elements shown in and described below with reference to Figure 12 may be one or 
several elements, such as one or more disks (which may, for example, be magnetic, 
optical, magneto-optical, etc.), the memory 654 and/or the mass memory 662 
described above with reference to Figure 7. Furthermore, in one embodiment where 

10 the generator, with which the machine readable storage medium shown in Figure 12 is 
associated, is a network computer, one or more of the elements of the machine 
readable storage medium may be stored at another digital processing system and 
downloaded to the generator. Furthermore, the elements described with reference to 
the machine readable storage medium may, at some point in time, be stored in a non- 

15 volatile mass memory (e.g., a hard disk). Conversely, at other times, the elements of 
the machine storage medium may be dispersed between different storage areas, such 
as DRAM, SRAM, disk, etc. 

Figure 12 shows a machine readable storage medium 740. In one 
embodiment, the machine readable storage medium is utilized, at least in part, by a 

20 digital processing system that generates hints or hint tracks, i.e., a generator, in 

accordance with one or more method(s) of the invention. The generator, as described 
with reference to Figure 8, may be integrated into a digital processing system that 
transmits media data according to the hint tracks, or may be, as described with 



reference to Figure 9, a digital processing system that creates and provides the hints to 
another digital processing system, such as a server, which utilizes the hints to 
packetize and transmit media data. 

As shown in Figure 12, the machine readable storage medium 740 typically 

5 includes a number of elements. For example, the machine readable storage medium 
740 includes software for providing operating system functionality to the generator, as 
depicted by a generator operating system (OS) 742. A network transnaission 
routine(s) 748 provides data communication functionality, such as routines, protocols, 
etc., to allow the generator to transmit and receive data via a data communication link. 

10 In addition, the machine readable storage medium 740 includes routines and 

data for creating hints associated with media transmission. As such, the machine 
readable storage medium 740 may optionally include information 750, which may 
provide information relating to one or more data communication protocols and media 
formats which may be necessary for creation of hints by a hint creation routine(s) 744. 

15 For example, the information 750 may include information relating to QuickTime 
movies, RTF, MPEG, etc. However, such information may, at least in part, be 
integrated into the hint creation routine 744 and/or be provided to the generator by a 
remote digital processing system. 

The hints created by the hint creation routine(s) 744 may be stored as created 

20 hints 746 and/or stored/transmitted elsewhere (e.g., at a remote digital processing 
device, which may be a server). The hints are hint tracks that are time-related for 
packetization and transmission of media data, which is also time-related (e.g., video, 
audio, video and audio, etc.). 
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Although the machine readable storage medium 740 is described with 
reference to a generator, the medium 740, at least in part, may be part of a number of 
types of digital processing systems, data storage media, etc. For example, the 
machme readable storage medium 740, at least in part, may be included as part of a 
5 server or other digital processing system. Furthermore, the machine readable storage 
medium 740, at least in part, may be included as part of a software utility on one or 
more disks or other machine readable media. 

Figure 13 is an example of a machine readable storage medium that may be 
accessed by a digital processing system, such as a server, according to one 
10 embodiment of the invention. It will be appreciated that the actual memory that stores 
the elements shown in and described below with reference to Figure 13 may be one or 
several elements, such as one or more disks (which may, for example be magnetic, 
optical, magneto-optical, etc.), the memory 654 and/or the mass memory 662 
described above with reference to Figure 7. Furthermore, in one embodiment where 
15 the server, with which the machine readable storage medium shown in Figure 13 is 
associated, is a network computer, one or more of the elements of the machine 
readable storage medium may be stored at another digital processing system and 
downloaded to the server. Furthermore, the elements described with reference to the 
machine readable storage medium may, at some point in time, be stored in a non- 
20 volatile mass memory (e.g., a hard disk). Conversely, at other times, the elements of 
the machine storage medium may be dispersed between different storage areas, such 
as DRAM, SRAM, disk, etc. 



Figure 13 shows a machine readable storage medium 760. In one 
embodiment, the machine readable storage medium is utilized, at least in part, to 
packetize media data for transmission on a data communication link in accordance widi 
one or more method(s) of the invention. The machine readable storage medium 760 

5 may be associated with a server, such as the server 694 described with reference to 
Figure 8, to include routines to create hint tracks and transmit media data according to 
the hint tracks. In another embodiment, the machine readable storage medium 760 
may be associated with a digital processing system, such as the server 700 described 
with reference to Figure 9, wherein a digital processing system, such a generator, 

10 includes routines to create hints, and the server, using the hints as processed by 
routines provided by the machine readable storage medium 760, may packetize and 
transmit media data. 

The machine readable storage medium 760 includes a number of elements. 
For example, the machine readable storage medium 760 includes software for 

15 providing operating system functionality to the server, as depicted by a server 
operating system (OS) 762. A network transmission routine(s) 768 provides data 
communication functionality, such as routines, protocols, etc., to allow the server to 
transmit and receive data via a data communication link. 

In addition, the machine readable storage medium 760 includes a media 

20 packetization routine 770 for packetizing media data, which may be time-related, 

based on hints, and which may also be packetized. Accordingly, the machine readable 
storage medium 760 includes a media data storage area 764 and a hint storage area 766 
to store media data (which may, for example, be QuickTime movies or other media 



-44- 

tracks) and hints (e.g., hint tracks), respectively. The hints may include hint tracks 
that are time-related for packetization and transmission of media data, which is also 
typically time-related (e.g., video, audio, video and audio). In one embodiment, the 
hint tracks are packetized separately from the media data packets. In one embodiment, 
5 hints include pointer information identifying media data (e.g., a particular packet(s)) 
which may be in a separate media file. 

Figure 14 is an example of a machine readable storage medium that may be 
accessed by a digital processing system, such as a receiving system or other digital 
processing system, according to one embodiment of the invention. It will be 

10 appreciated that the actual memory that stores the elements shown in and described 
below with reference to Figure 14 may be one or several elements, such as one or 
more disks (which may, for example be magnetic, optical, magneto-optical, etc.), the 
memory 654 and/or the mass memory 662 described above with reference to Figure 7, 
Furthermore, in one embodiment where the receiving system, with which the machine 

15 readable storage medium shown in Figure 14 is associated, is a network computer, 
one or more of the elements of the machine readable storage medium may be stored at 
another digital processing system and downloaded to the receiving system. 
Furthermore, the elements described with reference to the machine readable storage 
medium may, at some point in time, be stored in a non-volatile mass memory (e.g., a 

20 hard disk). Conversely, at other times, the elements of the machine storage medium 
may be dispersed between different storage areas, such as DRAM, SRAM, disk, etc. 

Figure 14 shows a machine readable storage medium 780. In one 
embodiment, the machine readable storage medium is utilized, at least in part, to 
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process media data packetized in accordance with one or more method(s) of the 
invention. The machine readable storage medium 780 may be associated with a 
receiving system, such as the client data processing system 682 described with 
reference to Figures 8 and 9, to include routines to present media data 
5 transmitted/received according to hints. Alternatively, the machine readable storage 
medium 780 may include media data having hints (e.g., hint tracks) embedded 
therein. Such embedded media data may be pre-packaged or generated by a routine 
stored on a machine readable storage medium, such as the machine readable storage 
medium 780. 

1 0 The machine readable storage medium 780 may include a number of elements. 

For example, the machine readable storage medium 780 includes software for 
providing operating system functionality to the receiving system, as depicted by a 
server operating system (OS) 772. A network transmission routine(s) 782 provides 
data conmiunication functionality, such as routines, protocols, etc, to allow the server 

15 to transmit and receive data via a data communication link. 

In addition, the ntiachine readable storage medium 780 includes a media 
presentation routine 778 for presenting media data packetized according to hints. 
Thus, the machine readable storage medium 780, and in particular, the media 
presentation routine 778, may include routines for decompression of audio and/or 

20 video data, displaying of video, and/or playing back audio, etc. Furthermore, the 
media presentation routine 778 typically provides handling of hints that are associated 
with the media data. In one embodunent, the hints are simply ignored as media is 
presented. 
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Optionally, the machine readable storage medium 780 may store media data 
that has been packetized according to hints as media data 774, and include a media 
data reassembly routine 776 to reassemble to the stored media data (e.g., to be 
presented, transmitted, etc.). 
5 Figure 15 is a diagram of a data storage and/or communication medium having 

stored/transported thereon media and hint information, according to one embodiment 
of the invention. A data storage and/or communication medium (medium) 800 is 
shown, which represents various types of transport and/or storage medium in which a 
media data packet 804 and a hint packet 806 packetized according to the present 

10 invention could be stored or transported. For example, the medium 800 may 
represent the mass memory 662 and/or the memory 654, described above with 
reference to Figure 7. The medium 800 may also represent a communication medium, 
such as the LAN bus 612 shown in Figure 6 or the data communication link 686 for 
transporting data/signals representing media and/or other information. 

1 5 The hint packet 806 and the media packet 804 may be integrated into one 

packet or be stored and/or transported separately, as depicted in Figure 15. 
Furthermore, the hint packet 806 and the media packet 804 may embody several types 
of formats, such as ones described herein or one associated with other media formats, 
network protocols, and/or digital processing device architecture. 

20 Provided below are some example formats of hints. It will be appreciated that 

the present invention, however, may be utilized with various types of network 
protocols, digital processing system architectures, media formats, etc., to provide 
transmission of time-based data. 
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alternatwe Embodiments 

While the invention has been described in terms of several embodiments and 
illustrative figures, those skilled in the art will recognize that the invention is not 
5 limited to the embodiments or figures described. In particular, the invention can be 
practiced in several alternative embodiments that provide packetization of time related 
media data. 

Therefore, it should be understood that the method and apparatus of the 
invention can be practiced with modification and alteration within the spirit and scope 
10 of the appended claims* The description is thus to be regarded as illustrative instead 
of limiting on the invention. 

A ppendix A - Packetization Hint Sample Description 

In one embodiment of the present invention, each hint track has a table of 
15 sample descriptions. Hint tracks typically have one sample description. The format 
for each sample description entry for a hint track, according to one embodiment of the 
present invention, is described below in Table 1. 



Table 1: Hint Track Sample Descri 


ption Format 


Hint Track Sample Description 


Bytes 


Sample description size 


4 


Data format 


4 


Reserved 


6 


Data reference index 


2 


Max packet size 


4 


Additional data table 


variable 



20 

The packetization hint header atom contains the following data elements: 
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10 



15 



20 



Field descriptions: 
Sample 

description size 



Data format 



Reserved 
Data reference 



Max packet size 



Additional Data 
Table 



25 



A 32-bit integer that specifies the number of bytes 

in the sample description. 

A 32-bit integer indicating the format of the hints 

stored in the sample data. Different formats may be 

defined for different hint types. The table below 

lists defined formats. 

Six bytes that are set to 0. 

A 16-bit integer that contains the index of the data 

index associated with the samples that use this sample 

description. Data references are stored in data 

reference atoms, 

A 32-bit integer indicating the maximum size of 
packets computed in this track. 
A table containing additional information needed 
on a per track basis. The values are tagged entries. 
There are no required entries. If an entry is not present 
in the table, a reasonable default may be used. 



The structure for the additional data table entries is shown in Table 2. 



Table 2: Additional Data Table Format 



Additional Data Table 


Bytes 


Entry length 


4 


Data type 


4 


Data 


Entry length - 8 
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The additional data table entries contain the following data elements: 
Field descriptions: 

Entry length A 32-bit integer indicating the length of the entire 

entry (includes 8 bytes for the length and type fields) 

in bytes. 

Data type A 32-bit integer indicating the meaning of the data 
in the entry. 

Data The data for this entry. The length of the data is 

indicated by the Data length field of the table. 

The following data tags may defined for several various types of data format 
types. Other tags may be created as required. 

Length Type Data Description 

9 *rely ' A 1 byte integer indicating whether or not this 

track should be sent over a reliable transport. 
Values of 0 and 1 are defined. If this tag is not 
present, it is assumed to have the value zero, 
indicating that it can be sent over unreliable 
transports, such as UDP. 

The following data format types are defined. New types may be defined as 
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Data Format Description 

' r t p ' The packetization hints for sending media over RTP for 

the specific media type and encoding as described by 
various IETF drafts of the Audio-Video Transport 
5 (AVT) working group. 

The following data tag is utilized in one embodiment for 'rtp' data. 
Length Type Data Description 

12 'tims' A 32-bit number indicating the RTP 

10 timescale. This tag is present in one 

embodiment for RTP data. 

The following data tags are optional for *rtp' data. 
Length Type Data Description 

12 'tsro' A 32-bit number indicating the 

15 random offset to add to the stored time 

stamp when sending the RTP packets. 
If this field is not present, a truly 
random number should be used, as 
per the RTP specification. The value of 
20 this field could be zero, indicating tiiat 

no random offset is to be added. 
10 'snro' A 16-bit number indicating the 

random offset to add to the sequence 
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number when sending the RTP 
packets. If this field is not present, a 
truly random number should be used, 
as per the RTP specification. The value 
5 of this field could be zero, indicating 

that no random offset is to be added. 
A ppendix B — Example hint track for RTP 

This section presents one example of a hint track format for streaming RTP 
from a QuickTime movie. 
10 In standard RTP, each media stream is typically sent as a separate RTP stream. 

Multiplexing is generally achieved by using IP's port-level multiplexing, not by 
interleaving the data from multiple streams into a single RTP session. Therefore each 
media track in the movie should have an associated RTP hint track. In one 
embodiment of the present invention, each hint track contains a track reference back to 
15 the media track which it is streaming. 

In this example, the packet size is determined at the time the hint track is 
created. Therefore, in the sample description for the hint track (a data structure which 
can contain fields specific to the 'coding' - which in this case is a protocol), the 
chosen packet size is indicated. In one example of the present invention, several RTP 
20 hint tracks are provided for each media track to provide different packet size choices. 
Other protocols may be parameterized as well. Similarly, the appropriate time-scale 
for the RTP clock is provided in the sample description below. 



-52- 

The hint track is related to its base media track by a single track reference 
declaration. (RTP does not permit multiplexing of media within a single RTP stream). 
The sample description for RTP declares the maximum packet size which this hint 
track will generate. Session description (SAP/SDP) information is stored in user-data 
5 atoms in the track. 

Each sample in the RTP hint track contains the instructions to send out a set of 
packets which must be emitted at a given time. The time in the hint track is emission 
time, not necessarily the media time of the associated media. 

In the following description the intemal stmcture of samples, which are media 
10 data, not meta data, in the terminology of this example is described, need not be 
structured as objects. 

In this example, each sample contains two areas: the instructions to compose 
the packets, and any extra data needed when sending those packets (e.g. an encrypted 

version of the media data). 

15 struct RTPsample { 

int ( 16 ) packet count ; 

RTPpacket packets [packetcount] ; 
byte [ ] extradata ; 

} 

20 Each RTP hint packet contains the information to send a single packet. In one 

embodiment, to separate media time from emission time, an RTP time stamp is 
specifically included, along with data needed to form the RTP header. In altemative 
embodiments, however, this is not the case. Other header information is typically 
supplied. A table of construction entries is constructed as follows: 
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struct RTPpacket { 

int(32) RTPtime; 
int (16) partialRTPheader; 
int(16) RTPsequenceseed; 
5 in t { 1 6 ) ent rye oiin t ; 

dataentry constructors [entrycount] ; 

} 

There are various forms of the constructor. Each constructor is 16 bytes, 

which may make iteration relatively simple. The first byte is a union discriminator: 

10 struct dataentry { 

int ( 8 ) entrytype ; 
switch entrytype { 

case immediate: 

int ( 8 ) bytecount ; 

15 int (8) bytestocopy [bytecount] ; 

case mediasample: 

int ( 8 ) reseirved [ 5 ] ; 

int (16) length ; 

int (32) mediasampleniunber; 
20 int (32) mediasainpleof fset; 

case hintsample: 

int ( 8 ) reserved [ 5 ] ; 

int (16) length; 
int (32) hintsamplenumber; 
25 int (32) hintsampleof f set ; 



The immediate mode permits the insertion of payload-specific headers (e.g. the 
30 RTP H.261 header). For hint tracks where the media is sent *in the clear*, the 

mediasample entry may specify the bytes to copy from the media track, by giving the 
sample number, data offset, and length to copy. For relatively complex cases (e.g. 
encryption or forward error correction), the transformed data may be placed into the 
hint samples, and then hintsample mode may be used, which would be provided from 
35 the extradata field in the RTPsample itself. 

In one example of the present invention, there is no requirement that 
successive packets transmit successive bytes from the media stream. For example, to 



-54- 

conform with RTP-standard packing of H.261, in one example of the present 
invention, a byte may be sent at the end of one packet and also at the beginning of the 
next (when a macroblock boundary falls within a byte). 

5 A ppendix C - Packetization Hint Sample Data for Data Format Vtp' 

This appendix provides a description of the sample data for the 'rtp' format, 
according to one embodiment of the invention. The 'rtp' format assumes that a server 
is sending data using Real Time Transport Protocol (RTF). This format assumes that 
the server knows about RTF headers, but does not require that the server know 
10 anything about specific media header, including media headers defined in various 
IETF drafts. 

In one embodiment of the present invention, each sample in the hint track will 
generate one or more RTF packets. Each entry in the sample data table in a hint track 
sample corresponds to a single RTF packet. Samples in the hint track may or may not 
15 correspond exactly to samples in the media track. In one embodiment of the present 
invention, data in the hint track sample is byte aligned, but not 32-bit aligned. 

Field descriptions: 

Entry count A 16-bit unsigned integer indicating the number of 

packet entries in the table. Each entry in the table 
20 corresponds to a packet. Multiple entries in a single 

sample indicate that the media sample had to be split 
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into multiple packets. A sample with an entry count of 
zero is reserved and if encountered, should be skipped. 

Packet entry table A variable length table containing packet entries. 
Packet entries are defined below. 

5 Additional data A variable length field containing data pointed to by the 

entries in the data table shown below by Table 3: 



Table 3 - Additional Data 



Packet Entry 


Bytes 


Relative packet transmission time 


4 


Flags 


4 


RTP header info 


2 


RTF sequence number 


2 


Entry count 


2 


Data table 


variable 



In one embodiment, the packet entry contains the following data elements: 

Field descriptions: 

relative packet A 32-bit signed integer value, indicating the time, 

transmission time 

in hint track's timescale, to send this packet relative 
to the hint sample's actual time. Negative values 
mean that the packet will be sent earlier than real 



10 



15 
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time, which is useful for smoothing the data rate. 
Positive values are useful for repeating packets at 
later times. Within each hint sample track, each 
packet time stamp is nondecreasing. 
5 flags A 32-bit field indicating certain attributes for this 

packet. 

The RTP header information field contains the following element: 
Field Bit # Description 

R 31 A 1 -bit number indicating that this is a 
1 0 repeat packet - the data has been defined in a previous 

packet. A server may choose to skip repeat packets to 
help it catch up when it is behind in its transmission of 
packets. AH repeated packets for a given packet care in 
the same hint sample. 



15 



All undefined bits (0-30) are reserved and are set to 
zero. 

RTP header info A 16-bit integer specifying various values to be set in 
the RTP header. 



20 



The RTP header information field contains the following elements: 
Field Bit# Description 
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P 2 A 1-bit number corresponding to the padding 

(P) bit in the RTP header. This bit may 
not be set, since a server that needed 
different packet padding may generally need to 
5 un-pad and re-pad the packet itself. 

X 3 A 1-bit number corresponding to the 

extension (X) bit in the RTP header. This bit 
may not be set, since a server that needs to send 
its own RTP extension may either not be able 
10 to, or may be forced to replace any extensions 

from the hint track. 

M 8 A 1-bit number corresponding to the marker 

(M) bit in the RTP header. 

payload 9-15 A 7-bit number corresponding to the pay load 
15 type 

type (PT) field of the RTP header. 

All undefined bits (0-1 and 4-7) are reserved and are set to zero. The 
location of the defined bits are in the same bit location as in the RTP 
20 header. 

RTP sequence A 16-bit integer specifying the RTP sequence number for 
number 

the packet. The RTP server adds a random offset to this 
sequence number before transmitting the packet This field 
25 allows re-transmission of packets, e.g., the same packet can 
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be assembled with the same sequence number and a different 
(later) packet transmission time. For example, a text sample 
with a duration of 5 minutes can be retransmitted every 10 
seconds so that clients that miss the original sample 
5 transmission (perhaps they started playing a movie in the 

middle) will be "refreshed" after a maximum of 10 seconds. 

Entry count A 16-bit unsigned integer specifying the number of entries in 
the data table. 

Data table A table that defines the data to be put in the pay load portion of 

10 the RTP packet. This table defines various places the data can 

be retrieved, and is shown by Table 4. 



Table 4 - Data Table 



Data table entry 


Bytes 


Data source 


1 


Data 


15 



The data source field of the entry table indicates how the other 15 bytes of the entry 
15 are to be interpreted. Values of 0 through 4 are defined. The various data table 
formats are defined below. Although there are various schemes, the entries in the 
various schemes are typically 16 bytes long. 

No-Op Data Mode 

The data table entry has the following format for no-op mode: 



I 
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Field description: 
Data source = 0 



A value of zero indicates that this data table entry is to 
be ignored. 



5 Lmnediate Data Mode 

The data table entry has the following format for immediate mode: 
Field description: 

Data source =1 A value of one indicates that the data is to be 

immediately taken from the bytes of data that follow. 
10 Immediate length An 8-bit integer indicating the number of bytes to take 

from the data that follows. Legal values range from 0 
to 14. 

Immediate data 14 bytes of data to place into the payload portion of the 
packet. Only the first number of bytes indicated by the 
1 5 immediate length field are used. 

Sample Mode 

The data table entry has the following format for sample mode: 
Field description: 

Data source =2 A value of two indicates that the data is to be taken from a 
20 track's sample data. 

Track ref index. A value that indicates which track the sample data will come 
from. A value of zero means that there is exactly one media 
track referenced, which is to be used. Values from 1 to 
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10 



15 



Bytes per 

compression 

block 



Samples per 
compression 
block 



Length 



127 are indices into the hint track reference atom entries, 
indicating from which original media track the sample is to 
be read. A value of -1 means the hint track itself, i.e., the 
sample from the same track as the hint sample cmrently 
being parsed is used. 

A 16-bit unsigned integer specifying the number of 
bytes that results from compressing the number of 
samples in the Samples per compression block field. A 
value of zero is equivalent to a value of 1. 

A 16-bit unsigned integer specifying the uncompressed 
samples per compression block. A value of zero is 
equivalent to a value of 1 . 

A 16-bit integer specifying the number of bytes in the 
sample to copy. 



Sample Number A 32-bit integer specifying sample number of the track. 



20 



Offset 



A 32-bit integer specifying the offset from the start of the 
sample from which to start copying. If referencing samples 
in the hint track, this will generally point into the Additional 
Data area. 
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ff the bytes per compression block and/or the samples per compression block 
is greater than 1, than this ratio is used to translate a sample number into an actual byte 
offset. This ratio mode is typically used for compressed audio tracks in QuickTime 
movies, such that: 

5 CB = NS * BPCB / SPCB 

wherein, 

CB = compressed bytes 
NS = number of samples 
BPCB = bytes per compression block 
10 SPCB = samples per compression block 



For example, a GSM compression block is typically 160 samples packed into 33 

bytes. Therefore, BPCB = 33 and SPCB = 160. The hint sample requests 33 bytes 

of data starting at the 161st media sample. Assuming that the first QuickTime chunk 

1 5 contains at least 320 samples, so after determining that this data will come from chunk 

1, and where chunk 1 starts, the ratio is utilized to adjust the offset into the file where 

the requested samples will be found: 

chunk„number = 1; /* calculated by walking the sample-to-chunk atom*/ 
first_samplejn„this_chunk = 1; /* also calculated from that atom*/ 
20 chunk_offset = chunk_offsets[chunk_number]; /* from the stco atom */ 

data_offset = (sample_number - first_sample_in.this_chunk) * BPP / SPP 
read_from_file(chunk_offset + data_offset, length); /* read our data */ 



Sample Description Mode 

The data table entry has the following format for sample description mode: 



10 
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Field description: 

Data source = 3 A value of three indicates that the data is to be taken from 
the media track's sample description table. 

Track ref index A value that indicates which track the sample data will come 
from. A value of zero means that there is exactly one hint 
track reference, which is to be used. Values from 1 to 127 
are indices into the hint track reference atom entries, 
indicating from which original media track the sample is to 
be read A value of -1 means the hint track itself, i.e., the 
sample description from the same track as the hint sample 
currently being parsed is utilized. 



15 



Reserved 



Length 



Sample 

description 

index 



Offset 



Four bytes that are set to zero. 

A 16-bit integer specifying the number of bytes in the 
sample to copy. 

A 32-bit integer specifying the index into the media's 
sample description table. 

A 32-bit integer specifying the offset from the start of the 
sample from which to start copying. 



20 



Additional data A variable length field containing data pointed to by hint 
track sample mode entries in the data table. 
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Appendix D — Example hint track format for MPEG-2 Transport 

This section presents one example of a simple track format for streaming 
MPEG-2 transport from a QuickTime movie holding elementary streams. 
5 An MPEG-2 transport stream is associated with a multiplex of one or more 

elementary streams. For this reason, an MPEG-2 transport hint track describes how 
to construct such a multiplex from one or more media tracks. There is not necessarily 
a one to one relationship between media tracks and MPEG-2 transport hint tracks. 
Each hint track may contain references to the elementary streams it represents. In one 

10 example of the present invention, a QuickTime file might contain multiple such hint 
tracks to describe different multiplexes. 

Packet size is generally not an issue, since all MPEG-2 transport packets are 
188 bytes in size. In one example of the present invention, each transport packet (in 
the MPEG-2 transport protocol) contains payload data from one media track. This 

15 allows for a relatively simple hint description for each transport packet. In one 

example of the present invention, each such hint describes which header data appears 
on each transport packet, and then points to the payload in the appropriate media track 
for the transport packet. For packets which do not correspond with a media track, 
such as PSI packets, the hint may describe 188 bytes of header data, and any media 

20 track reference may be considered irrelevant. For packets which do correspond with a 
media track, the header data may account for information such as transport headers, 
possible adaptation headers, and PES headers for transport packets that begin PES 
packets. 
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Reference is made to the MPEG-2 transport hint track in the Sample 
Description Atom (of type 'stsd'). This atom includes a sample description table, and 
the entries in this table differ based on the media type. In one example of the present 
invention, hint tracks begin with the structure shown in Table 1. The additional data 
5 table may hold entries with the structure shown in Table 2: 

In one example of the present invention, if the hint track is an MPEG-2 
transport hint track, the data format in the hint track sample description entry will be 
'm2t* and the max packet size will always be 188. In such a description entry, the 
types shown below in Tables 5-7 may be found in the additional data table: 
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Table 5 - Additional Data Table Entries 







Data f]p(srrintinn 

M-^<*V€t- U^OVI. l^t'EVrll 


8 


0x00000000 


Indicates there are no more entries in the table 


9 


'otyp' 


Describes how offsets are described in the 
hints. The one byte of data has values 
described below in figure This entry is 
mandatory in the additional data table. 


9 


'msns' 


Describes the size of media sample numbers. 
The one byte of data indicates how many bytes 
are used to specify media sample numbers. If 
this is not present, and media sample numbers 
are present in the sample data, the default 
value is 4 bytes. 


9 


*msos' 


Describes the size of media sample offsets. 
The one byte of data indicates how many bytes 
are used to specify media sample offsets. If 
this is not present, and media sample offsets 
are present in the sample data, the default 
value is 4 bytes. 


9 


*f0S2' 


Describes the size of file offsets. The one 
byte of data indicates how many bytes are used 
to specify file offsets within samples If this is 
not present, and file offsets are present in the 
sample data, the default value is 4 bytes. 


Variable 


*tmap' 


Describes an abbreviated mapping of media 
tracks. Each 5 byte entry maps a 4 byte track 
ID to a 1 byte track reference number. This 
limits any given transport mux to containing 
no more than 256 media tracks, but this should 
not be a limiting factor^ and this compression 
is useful in limiting the size of the hint track. 
The format of these 5 byte entries is specified 
below in figure B.5. This entry is mandatory 
in the additional data table. 



Table 6- 'otyp* Values In the Additional Data Table 



Value 


Description 


0 


Samples are described in terms of media samples 


1 


Samples are described in terms of file offsets 



Table 7 - Format of Entries in the *tmap' Additional Data Entry 



Length 


Description 


4 


Original Track ID 


1 


Abbreviated track reference number used in samples 



5 In one example of the present invention, each hint sample describes one 

transport packet. Each transport packet can be described as some amount of header 
data, followed by some amount of payload from one media track. Since MPEG-2 
transport packets are relatively small, a large number of hint samples may be 
generated, and thus, these samples preferably should be as small as possible. Several 

10 entries in the additional data table above may be used to minimize the size of samples, 
but such factors may make some of the fields in the sample entries variable in size. 

If the 'otyp' entry in the data table has the value 0, indicating that payload data 
is described in terms of media samples, hint samples may be of the following form 
shown in Table 8: 



15 
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Table 8 - Hint Sample Format Using Media Sample References 



Length 


Description 


1 


Track reference number of the media track holding the payload 
data for this packet. This can be mapped to a track ID using the 
^tmap^ entry in the additional data table. If the hint specifies 188 
bytes of immediate data, this field is irrelevant. 


1 


The length of the immediate data for the packet. Note that this 
must be 188 or less, since transport packets are 188 bytes in 
length. 


Variable 


Bytes of immediate data to be used as the header for the transport 
packet. The number of bytes is described by the previous field. 


Variable 


The media sample number to use for the payload data. The 
default size of this field is 4 bytes, but may be modified by the 
presence of an ^msns' entry in the additional data table. 


Variable 


The media sample offset to use for the payload data. The default 
size of this field is 4 bytes, but may be modified by the presence 
of an ^msos' entry in the additional data table. 



In one example of the present invention, it is not necessary to indicate the 
length of the payload data for the packet since in MPEG-2, this length is equal to 188 
minus the size of the header data for the packet. 

If the *otyp' entry in the data table has the value 1, indicating that payload data 
is described in terms of file offsets, hint samples may be of the following form shown 
in Table 9: 
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Table 9 



Length 


Description 


1 


Track reference number of the media track holding the payload 
data for this packet. This can be mapped to a track ID using the 
^tmap' entry in the additional data table. If the hint specifies 188 
bytes of immediate data, this field is irrelevant. 


1 


The length of the immediate data for the packet. Note that this 
must be 188 or less since transport packets are 188 bytes in 
length. 


Variable 


Bytes of immediate data to be used as the header for the transport 
packet. The number of bytes is described by the previous field. 


Variable 


The file offset where the payload data is located. This offset is 
in the file where the data for the media track is located. The 
default size of this field is 4 bytes, but may be modified by the 
presence of an ^fosz' entry in the additional data table. 



In one example of the present invention, hint samples may describe their 
offsets in terms of media samples or in terms of file offsets. Each of these has 
advantages and disadvantages. If hint samples specify payload in terms of media 
samples, they may be more resilient to additional editing of the file containing the 
media track, but may require additional processing for delivery. If hint samples 
specify payload in terms of file offsets, the payload data can be accessed relatively 
quickly, but any editing of the file containing the media track may invalidate the hints. 



A ppendix D — An example file 

Provided below is a relatively short (six frame) sample file, with some of the 
relatively less important fields and objects left out (marked here by ellipsis . and 
with some fictitious numbers to illustrate the overall structure of a file which is ready 
for streaming over RTP, according to one embodiment of the present invention. The 
media data has been left out; only the meta-data is shown. 
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moov — the entire movie meta-data 
mvhd — overall movie information 

TIME- SCALE 600 

DURATION 2792 

PREFERRED-RATE 1 

VOLUME 255 

MATRIX [[10 0] [0 1 0] [0 0 1]] 

NEXT-TRACK-ID 5 — tracks 1 to 4 are here 

trak — this is the video track 
tkhd 

TRACK- ID 1 
DURATION 2792 
LAYER 0 

MATRIX [[10 0] [0 1 0] [0 0 1]] 

WIDTH 176 
HEIGHT 144 
mdia 
mdhd 

TIME- SCALE 600 
DURATION 2722 

hdlr — we use the basic video media handler 

TYPE itihlr 

SUBTYPE vide 

MANUFACT appl 

NAME Apple Video Media Handler 
minf 
vmhd 

hdlr — basic 'alias' disk data handler gets the 

TYPE dhlr 
SUBTYPE alls 
MANUFACT appl 

NAME Apple Alias Data Handler 

dinf 
dref 

ENTRY-COUNT 1 

REFS [Pointer to this file] 

stbl — the complete sample table 
stsd — the sample description (s) 
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ENTRY-COUNT 1 

DESCRIPTIONS [video sample description] 

stts — convert time to sample 



ENTRY-COUNT 
TIMETOSAMPLE 



((1 200) 

(1 251) 

(1 479) 

(1 531) 

(1 1022) 

(1 239)) 

stss — 'sync' or key sample numbers 

ENTRY-COUNT 1 
SYNCSAMPLES (1) 
stsc — sample to chunk 



count, duration 



ENTRY-COUNT 1 
SAMPLETOCHUNK {(1 1 1)) 

— 1st chunk, samples /chunk, desc. 
stsz — sample sizes 



number 



di fferent 



DEFSAMPLESIZE 

ENTRY-COUNT 
SAMPLESIZES 



no default size, all 



6 

(664 
616 
1176 
1304 
2508 
588) 

stco — chunk offsets into file 



ENTRY-COUNT 
CHUNKOFFSETS 



6 

(4743 
5407 
8010 
12592 
17302 
25268) 



trak — this is the sound track 
tkhd 



TRACK- ID 
DURATION 



2 

2792 



VOLUME 



mdia 
mdhd 



GSM] 
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TIME- SCALE 8000 
DURATION 37280 
LANGUAGE US English 

hdlr — handled by the basic sound handler 

TYPE mhlr 

SUBTYPE soun 

MANUFACT appl 

NAME Apple Soiind Media Handler 
minf 
smhd 

BALANCE 0 
hdlr — data fetched by usual disc data handler 

TYPE dhlr 

SUBTYPE alis 

MANUFACT appl 

NAME Apple Alias Data Handler 
dinf 
dref 

ENTRY-COUNT 1 

REFS [Pointer to this file] 

stbl — sample table for the soimd 
stsd — sample descriptions 

ENTRY-COUNT 1 

DESCRIPTIONS [Sound sample description, incl 

stts — time to sample table 

... sound is measured by uncompressed samples 

ENTRY-COUNT 1 

TIMETOSAMPLE ((37280 1)) 

stsc 

ENTRY-COUNT 2 
SAMPLETOCHUNK ({1 4000 1) 

(10 1280 D) 
— first chunk, samples /chunk, desc. number 

stsz 

DEFSAMPLESIZE 1 — all samples same size 

ENTRY-COUNT 37280 
stco — chunk offset table 



ENTRY-COUNT 



10 



* 
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CHUNKOFFSETS (3093 

3918 
6023 
9186 
10915 
13896 ...) 

trak — the RTP hints for the video track 
tkhd 

TRACK- ID 3 
DURATION 2792 

tref 

hint — references the video track 

TRACKIDS { 1 ) 

mdia 
mdhd 

TIME- SCALE 600 

DURATION 2792 

hdlr — is 'played' by the hint media handler 

TYPE mhlr 
SUBTYPE hint 
MANUFACT appl 

NAME hint media handler 

minf 
gmhd 

hdlr — if played, the regular disc handler would fetch data 

TYPE dhlr 

SUBTYPE alis 

MANUFACT appl 

NAME Apple Alias Data Handler 
dinf 
dref 

ENTRY- COUNT 1 

REFS [Pointer to this file] 
stbl — samples describe packets 
stsd 

ENTRY-COUNT 1 

DESCRIPTIONS [hint sample description] 
stts — one packet per frame for video 



ENTRY-COUNT 



6 
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TIMETOSAMPLE { ( 1 270) 

(1 251) 

(1 479) 

(1 531) 

(1 1022) 

(1 239)) 

stss -- key sample derive from video 

ENTRY-COUNT 1 
SYNCSAMPLES (1) 
stsc — sample to chxmk table 

ENTRY-COUNT 1 
SAMPLETOCHUNK {(1 1 1)) 

stsz — sample sizes (packet instructions) 

DEFSAMPLESI2E 0 
ENTRY-COUNT 6 
SAMPLESIZES (52 

52 

52 

52 

102 

52) 

stco — chunk offsets 

ENTRY-COUNT 6 
CHUNKOFFSETS {6848 

6900 

10011 

14721 

20635 

25856) 

udta — track is named for ease of idientification 
name 

NAME Hinted Video Track * 

trak -- the RTP hints for the sound track 
tkhd 

TRACK- ID 4 

tref — references the sound track 
hint 

TRACKIDS (2) 

mdia 
mdhd 

TIME- SCALE 8000 
DURATION 37120 

hdlr 
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TYPE 

SUBTYPE 

MANUFACT 



NAJyrE 
minf 

hdlr 



mhlr 
hint 
appl 

hint media handler 



TYPE 

SUBTYPE 

MANUFACT 



NAME 
dinf 
dref 



dhlr 
alis 
appl 

Apple Alias Data Handler 



ENTRY-COUNT 
REFS 

stbl 
stsd 



[Pointer to this file] 



ENTRY-COUNT 
DESCRIPTIONS 
stts — time to sample 

ENTRY-COUNT 
TIMETOSAMPLE 



stsc 



[hint sample description] 



4 



({1 960) 

(7 4000) 

(1 1120) 

(1 7040)) 



ENTRY-COUNT 
SAMPLETOCHUNK 
Stsz 



1 

(d 



1 D) 



DEFSAMPLESI2E 

ENTRY-COUNT 

SAMPLESI2ES 



0 

10 

(206 
852 
852 
852 
852 
852 



stco 



ENTRY-COUOT 
CHUNKOFFSETS 



10 

(6952 
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7158 
10063 
11740 
14773 

16450 ...) 

udta 

NAME Hinted Sound Track 



SI 
ffl 

m 
rii 

rii 
rii 

iy 



f 
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CLAIMS 

What is claimed is: 



1 1 . A method implemented by a digital processing system for processing media 

2 data, said method comprising: 

3 creating on a first digital processing system a set of data to indicate how to 

4 transmit a time related sequence of media data according to a 

5 transmission protocol; and 

6 storing said set of data on a storage device coupled to the first digital 

7 processing system, wherein said set of data is a time related sequence 

8 of data associated with and separate from said time related sequence of 

9 media data. 

12, A method as in claim 1 wherein said set of data is stored as a track of 

2 indicating data, and wherein said transmission protocol comprises a packet data 

3 protocol. 

13. A method as in claim 1 further comprising: 

2 determining a format of said time related sequence of media data before 

3 creating said set of data; 

4 determining said transnodssion protocol before creating said set of data, 

5 wherein said transmission protocol is used to transmit said time related 

6 sequence of media data which has said format. 
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1 



4. 



A method as in claim 1 further comprising: 



2 



transmitting packets of data representing said time related sequence of media 



3 



data according to said transmission protocol. 



1 



5. 



A method as in claim 4 further comprising: 



2 



transmitting said set of data to a second digital processing system, which 



3 



second digital processing system, in response to receiving said set of 
data, generates said packets of data. 



4 



1 6 . A method as in claim 4 wherein for each of said packets, said set of data refers 

2 to data in at least one of a sequence of image data or a sequence of audio data 

3 associated with said time related sequence of media data. 

1 7 . A method as in claim 5 wherein said first digital processing system provides 

2 said set of data to a server digital processing system which stores said set of data and 

3 transmits said packets of data to a receiving digital processing system. 

1 8 , A machine readable medium containing executable program instructions, 

2 which when executed on a digital processing system cause the digital processing 

3 system to perform a method comprising: 

4 retrieving a set of data which indicates how to transmit a time related sequence 

5 of media data according to a transmission protocol; 
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6 



transmitting data representative of said time related sequence of media data 
according to said set of data, wherein said set of data is a time related 



7 



8 



sequence of data associated with and separate from said time related 



9 



sequence of media data. 



1 9 , The machine readable medium of claim 8, wherein said set of data is stored as 

2 a track of indicating data, and wherein said transmission protocol comprises a packet 

3 data protocol 

1 10. The machine readable medium of claim 8, wherein execution of said 

2 executable program instructions further cause said digital processing system to 

3 perform the method comprising: 

4 determining a format of said time related sequence of media data; 

5 determining said transmission protocol, wherein said transmission protocol is 

6 used to transmit said time related sequence of media data which has 

7 said format. 

1 11. The machine readable medium of claim 10, wherein execution of said 

2 executable program instructions further cause said digital processing system to 

3 perform the method comprising: 

4 transmitting packets of data representing said time related sequence of media 

5 data according to said transmission protocol. 
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1 12. The machine readable medium of claim 1 1 , wherein for each of said packets, 

2 said set of data refers to data in at least one of a sequence of image data or a sequence 

3 of audio data associated with said time related sequence of media data. 

1 13. The machine readable medium of claim 8, comprising a magnetic storage area, 

2 wherein at least one of said executable program instmctions and said time related 

3 sequence of media data is stored in said magnetic storage area. 

1 14. The machine readable medium of claim 8, comprising an optical storage area, 

2 wherein at least one of said executable program instmctions and said time related 

3 sequence of media data is stored in said optical storage area. 

1 15. The machine readable medium of claim 8, comprising an electronic storage 

2 area, wherein at least one of said executable program instmctions and said time related 

3 sequence of media data is stored in said electronic storage area. 

1 16. An apparatus comprising: 

2 a first digital processing system comprising a first processor to generate a set 

3 of data associated with transmission of a time related sequence of 

4 media data according to a transmission protocol, wherein said set of 

5 data is a time related sequence of data associated with and separate 

6 from said time related sequence of media data. 
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1 17. The apparatus of claim 1 6, further comprising: 

2 a second digital processing system, coupled to said first digital processing system, to 
' 3 receive said set of data from said first digital processing system, said second 

4 processor comprising: 

5 a second processor; 

6 a first storage area to store said media data; and 

7 a second storage area to store said set of data. 

1 18. The apparatus of claim 17, wherein said second digital processing system is 

2 coupled to a data communication link to provide packets of data representing said time 

3 related sequence of media data according to said transmission protocol. 

1 19. The apparatus of claim 1 8, wherein for each of said packets, said set of data 

2 refers to data in at least one of a sequence of image data or a sequence of audio data 

3 associated with said time related sequence of media data. 

1 20 . A computer readable medium comprising: 

2 a time related sequence of media data; 

3 a set of data which, when processed by a digital processing system, indicates 

4 to said digital processing system how to transmit said time related 

5 sequence of media data according to a transmission protocol, wherein 

6 said set of data is a time related sequence of data associated with and 

7 separate from said time related sequence of media data. 
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1 21. The computer readable medium of claim 20, wherein said set of data is stored 

2 as a track of indicating data, and wherein said transmission protocol comprises a 

3 packet data protocol. 



1 22. The computer readable medium of claim 20, further comprising: 

2 a first set of instructions to cause a digital processing system to determine a 

3 format of said time related sequence of media data; 

4 a second set of instructions to cause said digital processing system to 

5 determine said transmission protocol, wherein said transmission 

6 protocol is used to transmit said time related sequence of media data 

7 which has said format. 

1 23 . The computer readable medium of claim 22, wherein said set of data is stored 

2 as a track of indicating data, and wherein said transmission protocol comprises a 

3 packet data protocol. 

1 24. The computer readable medium of claim 21 , ftuther comprising a set of 



2 instructions to cause a digital processing system to generate packets representing said 

3 time related sequence of media data, wherein for each of said packets, said set of data 

4 refers to data in at least one of a sequence of image data and a sequence of audio data 

5 associated with said time related sequence of media data. 
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1 25 . The computer readable mediiun of claim 20, comprising a magnetic storage 

2 area, wherein at least one of said time related sequence of media data and said set of 

3 data is stored in said magnetic storage area, 

1 26. The computer readable medium of claim 20, comprising an optical storage 

2 area, wherein at least one of said time related sequence of media data and set of 

3 instructions is stored in said optical storage area. 

1 27 . The computer readable medium of claim 20, comprising an electronic storage 

2 area, wherein at least one of said time related sequence of media data and said set of 



3 data is stored in said electronic storage area. 

1 28 . A computer readable medium containing executable computer program 

2 instructions, which when executed on a &st digital processing system cause the fnrst 

3 digital processing system to perform a method comprising: 

4 generating a set of data to indicate a method to transmit a time related sequence 

5 of media data according to a transmission protocol, wherein said set of 

6 data is a time related sequence of data associated with and separate 

7 from said time related sequence of media data; and 

8 storing said set of data. 
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1 29. The computer readable medium of claim 28, wherein said set of data is stored 

2 as a track of indicating data, and wherein said transmission protocol comprises a 

3 packet data protocol. 



1 30, The machine readable medium of claim 28, wherein said executable program 

2 instructions further cause the first digital processing system to perform the method 

3 comprising: 

4 determining a format of said time related sequence of media data; 

5 determining said transmission protocol, wherein said transmission protocol is 

6 used to transmit said time related sequence of media data which has 

7 said format. 



1 31. The machine readable medium of claim 28, wherein said executable program 

2 instructions further cause the first digital processing system to perform the method 

3 comprising: 

4 generating packets of data representing said time related sequence of media 

5 data according to said transmission protocol; and 

6 transmitting said packets to a second digital processing system. 



1 
2 
3 



32. The machine readable medium of claim 28, wherein said executable program 
instructions further cause the digital processing system to perform the method 
comprising: 
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4 transmitting said set of data to a second digital processing system, wherein 

5 said second digital processing system utilizes said set of data to 

6 generate packets of data representing said time related sequence of 

7 media data according to said transmission protocol. 

1 33. The machine readable medium of claim 3 1 , wherein for each of said packets, 

2 said set of data refers to data in at least one of a sequence of image data and a sequence 

3 of audio data associated with said time related sequence of media data. 

1 34 . The machine readable medium of claim 22, wherein for each of said packets, 

2 said set of data refers to data in at least one of said sequence of image data and said 

3 sequence of audio data, 

1 35. The machine readable medium of claim 32, wherein said second digital 

2 processing system, in response to said set of data, transmits said packets of data to 

3 another digital processing system. * 

1 36. An apparatus for processing media data, said apparatus comprising: 

2 a first means for generating a set of data associated with transmission of a time 

3 related sequence of media data according to a transmission protocol, 

4 wherein said set of data is a time related sequence of data associated 

5 with and separate from said tune related sequence of media data; and 

6 a second means for storing said first set of data. 
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1 37 . The apparatus of claim 36, further comprising: 

' 2 a third means for transmitting packets of data representing said time related 

3 sequence of media data. 

1 38. The apparatus of claim 37, wherein said set of data identifies at least a portion 

2 of said packets of data. 

1 39. The apparatus of claim 37, wherein said set of data provides at least a portion 

2 of the information included in said packets of data. 

1 40. The apparatus of claim 37, further comprising: 

2 a third means for transmitting said set of data to a server means, said server 

3 means having means for generating packets of data representing said 

4 time related sequence of media data for transmission to a receiver 

5 means. 

1 41. A method of processing media data, said method comprising: 

2 storing a time related sequence of media data; 

3 storing a set of data to enable a first digital processing system to generate, 

4 according to a transmission protocol, data packets representing said 

5 time related sequence of media data, wherein said set of data is a time 



1 
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6 related sequence of data associated with said time related sequence of 

7 media data. 

1 42 . The method of claim 4 1 , wherein said set of data provides at least a portion of 

2 the information included in said data packets, 

1 43, The method of claim 41, wherein said set of data identifies at least a portion of 

2 the information included in said data packets. 

1 44 . The method of claim 4 1 , further comprising: 

2 generating said set of data at a second digital processing system; 

3 said second digital processing system transmitting said set of data to said first 

4 digital processing system; and 

5 said first digital processing system generating said data packets in response to 

6 receiving said set of data. 

1 45 . The method of claim 44, further comprising: 

2 said fn-st digital processing system transmitting said data packets to another 

3 digital processing system for presentation as a media object. 



1 

2 



46. A method implemented by a digital processing system for processing media 
data, said method comprising: 
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3 generating on a first digital processing system a first time related sequence of 

4 data to indicate how to transmit a second time related sequence of data 

5 according to a transmission protocol, wherein said second time related 

6 sequence of data is associated with time-based media, and wherein said 

7 first time related sequence of data is associated with said second time 

8 related sequence of data; and 

9 storing said first time related sequence of data. 

1 47 . A method as in claim 46, wherein said first time related sequence of data is 

2 stored as a track of indicating data, and wherein said transmission protocol comprises 

3 a packet data protocol 

1 48 . A method as in claim 46, further comprising: 

2 determining a format of said second time related sequence of data prior to 

3 generating said first time related sequence of data; and 

4 determining said transmission protocol prior to generating said first time 

5 related sequence of data, wherein said transmission protocol is used to 

6 transmit said second time related sequence of data which has said 

7 format. 

1 49 . A method as in claim 46, further comprising: 

2 transmitting packets of data representing said second time related sequence of 

3 data according to said transmission protocol. 
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1 



50. A method as in claim 49, further comprising: 



2 



transmitting said first time related sequence of data to a second digital 
processing system, which second digital processing system, in 
response to receiving said &st time related sequence of data, generates 



3 



4 



5 



said packets of data. 



1 51. A method as in claim 49, wherein for each of said packets, said first time 

2 related sequence of data refers to at least one of a sequence of image data or a 

3 sequence of audio data associated with said second time related sequence of data. 

1 52. A method as in claim 50, wherein said &st digital processing system provides 

2 said first time related sequence of data to a server digital processing system which 

3 stores said first time related sequence of data and transmits said packets of data to a 

4 receiving digital processing system. 

1 53 . A method as in claim 50, further comprising presenting said time related 

2 sequence of media data on at least one of said first digital processing system and said 

3 second digital processing system. 

1 54. A method as in claim 46, wherein said second time related sequence of data is 

2 stored on a read-only memory (ROM). 
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1 55 . A method as in claim 54, wherein said read-only memory (ROM) comprises a 

2 optical storage medium. 

1 56. A method as in claim 54, wherein said second time related sequence of data is 

2 packetized according to said first time related sequence of data without performing at 

3 least one of a storing and a formatting operation on said second time related sequence 

4 of data. 
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ABSTRACT OF THE DTSCLOSTTRF 

Methods and apparatuses for processing media data for transmission in a data 
communication medium. A set of data indicates how to transmit a time related 
sequence of media data according to a transmission protocol. The set of data, includes 
a time related sequence of data which is associated with the time related sequence of 
media data. The set of data may be utilized by a digital processing system to transmit 
the time related sequence of media data (e.g., by packets generated according to the 
transmission protocol and the set of data). 
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(City, State) (Country) 

Post Office Address 955 Escalon Avenue #515 



Sunnyvale. CA 94086 



Rev. 08/05/98 CIPVer.2 



-3- 



Full Name of Fourth/Joint Inventor Alaou Periyannan 



Inventor's Signature Date, 



Residence Fremont , California Citizenship India 

(City, State) (Country) 

Post Office Address 34113 Finnigan Terrace 

Fremont, CA 94555 



Full Name of Fifth/Joint Inventor David W. Singer 
Inventor's Signature _ 



Residence San Francisco. California Citizenship United Kingdom 

(City, State) (Country) 

Post Office Address 268 Wawona Street 

San Francisco. CA 94127 
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Title 37, Code of Federal Regulations, Section 1.56 
Duty to Disclose infomiation Material to Patentability 



(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the nnost effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all infomnation material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in dealing 
with the Office, which includes a duty to disclose to the Office all information known to that individual to be 
material to patentability as defined in this section. The duty to disclosure information exists with respect to 
each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the infomiation is not material to the patentability of any claim remaining 
under consideration in the application. There is no duty to submit information which is not material to the 
patentability of any existing claim. The duty to disclosure all information known to be material to patentability 
is deemed to be satisfied if all information known to be material to patentability of any claim issued in a patent 
was cited by the Office or submitted to the Office in the manner prescribed by §§1.97(b)-(d) and 1.98. 
However, no patent will be granted on an application in connection with which fraud on the Office was practiced 
or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. The Office 
encourages applicants to carefully examine: 

(1) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest infomiation over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material infomiation 
contained therein is disclosed to the Office. 

(b) Under this section, infomiation is material to patentability when it is not cumulative to 
infonnation already of record or being made or record in the application, and 

(1) it establishes, by itself or in combination with other infomiation, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the infomiation compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each temi in the claim its 
broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1 ) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attomey, agent or inventor may comply with this section by 
disclosing information to the attomey, agent, or inventor. 
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Attorney's Docket No.: 04860.P2207X 



PATENT 



DECLARATION AND POWER OF ATTORNEY FOR PATENT APPUCATION 
(CONTINUATION-IN-PART^ 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below, next to my name. 

I believe I am the original, first, and sole inventor (if only one name is listed below) or an 
original, first, and joint inventor (if plural names are listed below) of the subject matter 
which is claimed and for which a patent is sought on the invention entitled 

METHOD AND APPARATUS FOR MEDIA DATA TRANSMISSION 

the specification of which 

is attached hereto. 

X was filed on August 25. 1998 jas 

United States Application Number 09/140.173 

or PCT international Application Number 

and was amended on - 

(if applicable) 

I hereby state that i have reviewed and understand the contents of the above-identified 
specification, including the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose all information known to me to be material to patentability 
as defined in Title 37, Code of Federal Regulations, Section 1 ,56, 

I hereby claim foreign priority benefits under Title 35, United States Code, Section 
119(a)-(d), of any foreign application(s) for patent or inventor's certificate listed below 
and have also identified below any foreign application for patent or inventor's certificate 
having a filing date before that of the application on which priority is claimed: 

Priority 

Prior Foreign Application(s) Claimed 



(Number) (Country) (Day/Month/Year Filed) Yes No 



(Number) (Country) (Day/Month/Year Filed) Yes No 



(Number) (Country) (Day/MonthA'ear Filed) Yes No 
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I hereby claim the benefit under title 35, United States Code, Section 119(e) of any United 
States provisional application(s) listed below 

60/071.566 Januan^lS. 1998 

(Application Number) Filing Date 



(Application Number) Filing Date 



I hereby claim the benefit under Title 35. United States Code. Section 120 of any 
United States appiication(s> listed below and, insofar as the subject matter of each 
of the claims of this application is not disclosed in the prior United States 
application in the manner provided by the first paragraph of Title 35, United States 
Code, Section 112, I acknowledge the duty to disclose all information known to me 
to be material to patentability as defined in Title 37, Code of Federal Regulations, 
Section 1.56 which became available between the filing date of the prior 
application and the national or PCT international filing date of this application: 



(Application Number) Filing Date (Status -- patented, 

pending, abandoned) 



(Application Number) Filing Date (Status -* patented, 

pending, abandoned) 



I hereby appoint Farzad E. Amini, Reg. No. P42,261; Aloysius T. C. AuYeung, Reg. No. 35.432; 
Amy M. Armstrong, Reg. No. P42,265; William Thomas Babbitt, Reg. No. 39,591; Carol F. Barry, 
Reg. No. P41,600; Jordan Michael Becker, Reg. No. 39,602; Bradley J. Bereznak, Reg. No. 33,474; 
Michael A. Bernadicou, Reg. No. 35,934; Roger W. Blakely, Jr., Reg. No. 25.831; Gregory D. 
Caldwell, Reg. No. 39.926; Kent M. Chen. Reg. No. 39.630; Lawrence M. Cho, Reg. No. 39,942; 
Yong S. Choi, Reg. No. P43,324; Thomas M. Coaster, Reg. No. 39,637; Roland B. Cortes, Reg. No. 
39,152; Barbara Bokanov Courtney, Reg. No. P42,442; Michael Anthony DeSanctis, Reg. No. 
39,957; Daniel M. De Vos, Reg. No. 37.813; Tarek N. Fahmi, Reg. No. 41,402; James Y. Go, 
Reg. No. 40,621; Richard Leon Gregory, Jr., P42,607; Dinu Gruia, Reg. No. P42,996; David R. 
Halvorson, Reg, No. 33,395; Thomas A, Massing, Reg. No. 36.159; Phuong-Quan Hoang, 
P41,839; Willmore F. Holbrow III, Reg. No. P41,845; George W Hoover II, Reg. No. 32,992; 
EricS, Hyman, Reg. No. 30,139; Dag H. Johansen, Reg. No. 36,172; William W. KIdd, Reg. No. 
31,772; Tim L. Kitchen, Reg. No. P41,900; Michael J. Mallie, Reg. No. 36,591; Andre L 
Marais, under 37 C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. P42,879; Darren J. Milllken, 
P42,004; Thinh V. Nguyen, P42,034; Kimberley G. Nobles, Reg. No. 38,255; Michael A. 
Proksch. Reg. No. P43,021; Babak Redjaian, P42,096; James H. Salter, Reg. No. 35,668; 
William W. Schaal, Reg. No. 39.018; James C. Scheller, Reg. No. 31,195; Anand Sethuraman, 
Reg. No. P43,351; Charles E. Shemwell, Reg. No. 40,171; Maria McCormack Sobrino, Reg. No. 
31,639; Stanley W, Sokoloff, Reg. No. 25,128; Allan T, Sponseller, Reg. No. 38,318; Geoffrey 
T. Staniford, P43,151; Judith A. Szepesi, Reg. No. 39,393; Vincent P. Tassinari, Reg. No. 
P42,179; Edwin H. Taylor, Reg.- No. 25,129; George G. C. Tseng, Reg. No. 41,355; Lester J. 
Vincent, Reg. No. 31.460; John Patrick Ward, Reg. No. 40,216; Stephen Warhola. Reg. No. 
P43,237; Ben J. Yorks. Reg, No. 33,609; and Norman Zafman, Reg. No. 26,250; my attorneys; 
and Robert Andrew Diehl, Reg. No. 40,992; my patent agent, of BLAKELY, SOKOLOFF, TAYLOR & 
ZAFMAN LLP. with offices located at 12400 Wilshire Boulevard, 7th Floor, Los Angeles, 
California 90025, telephone (310) 207-3800; and James R. Thein, Reg. No. 31,710, my 
patent attorney; with full power of substitution and revocation, to prosecute this application and 
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to transact all business in the Patent and Trademark Office connected herewith. I also hereby 
appoint Mark Aaker, Reg. No. 32,667; Richard Liu, Reg. No. 34,377; Helene Plotka Workman 
Reg. No. 35,981; Edward W. Scott, IV, Reg. No, 36,000; and Nancy R. Simon, Reg. No. 36,930- 
my attorneys; of APPLE COMPUTER, INC., located at 1 Infinite Loop, MS: 38-PAT, Cupertino, ' 
California 95014, telephone (408)974-9453, with full power of substitution and revocation, 
to prosecute this application and to transact all business in the Patent and Trademark Office 
connected herewith. 



Send correspondence to James C. Scheller. Jr. , BLAKELY, SOKOLOFF, TAYLOR & 

(Name of Attorney or Agent) 
ZAFMAN LLP, 12400 Wilshire Boulevard, 7th Floor, Los Angeles, California 90025 and 

direct telephone calls to James C. Scheller. Jr. , (408) 720-8598. 

(Name of Attorney or Agent) 

I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 

Full Name of Sole/First Inventor Anne Jones 



Inventor's Signature Date. 



Residence Redwood Citv. California Citizenship United States 

(City, State) (Country) 

Post Office Address 3817 Hamilton Way 

Redwood Citv. CA 94062 



Full Name of Second/Joint Inventor Jav Geagan 



Inventor's Signature Date. 



Residence San Jose. California Citizenship United States 

(City, State) (Country) 

Post Office Address 5475 Prospect Road #212 

San Jose. CA 95129 



Full Name of Third/Joint Inventor Kevin L. Gong 



Inventor's Signature Date 



Residence Sunnyvale. California Citizenship United States 

(City, State) (Country) 

Post Office Address 955 Escalon Avenue #515 

Sunnwale. CA 94086 
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Full Name of Fourth/Joint Inventor Alaou Periyannan 

Inventor's Signature P /Ik^^i^^-^^ Date ^ / ^< 1^1 

Residence Fremont, California Citizenship India 

(City, State) (Country) 

Post Office Address 34113 Finnigan Terrace 

Fremont, CA 94555 



Full Name of Fifth/Joint Inventor David W. Singer 

Inventor's Signature Date 

Residence San Francisco, California Citizenship United Kingdom 

(City, State) (Country) 

Post Office Address 268 Wawona Street 

San Francisco. CA 94127 
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Title 37, Code of Federal Regulations, Section 1.56 
Duty to Disclos e Information Material to Patentahility 

(a) A patent by its very nature is affected with a public interest. The public interest is best served 
and the most effective patent examination occurs when, at the time an application is being examined the ' 
Office IS aware of and evaluates the teachings of all information material to patentability Each individual 
^^i^ul^^^i^**^ prosecution of a patent application has a duty of candor and good faith in dealina 
with the Office, which includes a duty to disclose to the Office all infomiation known to that individual to be 
matenal to patentability as defined in this section. The duty to disclosure information exists with respect to 
each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the infonnation is not material to the patentability of any claim remainina 
under consideration In the application. There is no duty to submit information which is not material to the 
patentability of any existing claim. The duty to disclosure all infomiation known to be material to patentabilitv 
IS deemed to be satisfied if all infomiation known to be material to patentability of any claim issued in a oatent 
was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-{d) and 1 98 
However, no patent will be granted on an application in connection with which fraud on the Office was practiced 
or attempted or the duty of disclosure was violated through bad faith or intentional misconduct The Office 
encourages applicants to carefully examine: 

(1 ) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

. . . information over which individuals associated with the filing or prosecution of a 

patent application believe any pending claim patentably defines, to make sure that any material infomiation 
contained therein is disclosed to the Office. 

(b) Under this section, infonnation is material to patentability when it is not cumulative to 
infonnation already of record or being made or record in the application, and 

(1) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the infomiation compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim its 
broadest reasonable construction consistent with the specification, and before any consideration is aiven to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved In the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section bv 
disclosing information to the attorney, agent, or inventor. ^ 
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Attorney's Docket No.: 04860.P2207 patent 
DECLARATION AND POWER OF ATTORNEY FOR PATENfT APPLICATiON 



As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below, next to my name. 

I believe I am the original, first, and sole inventor (if only one name is listed below) or an 
original, first, and joint inventor (if plural names are listed below) of the subject matter 
which is claimed and for which a patent is sought on the invention entitled 

METHOD AND APPARATUS FOR MEDIA DATA TRANSMISSION 

the specification of which 

X is attached hereto. 

was filed on as 

United States Application Number 

or PCT International Application Number 

and was amended on . 

(if applicable) 

I hereby state that I have reviewed and understand the contents of the above-identified 
specification, including the claim(s), as amended by any amendment referred to above. I do not 
know and do not believe that the claimed invention was ever known or used in the United States 
of America before my invention thereof, or patented or described in any printed publication in 
any country before my invention thereof or more than one year prior to this application, that 
the same was not in public use or on sale in the United States of America more than one year 
prior to this application, and that the invention has not been patented or made the subject of an 
inventor's certificate issued before the date of this application in any country foreign to the 
United States of America on an application filed by me or my legal representatives or assigns 
more than twelve months (for a utility patent application) or six months (for a design patent 
application) prior to this application. 

I acknowledge the duty to disclose all information known to me to be material to patentability as 
defined in Title 37, Code of Federal Regulations, Section 1.56. 

I hereby claim foreign priority benefits under Title 35, United States Code. Section 119(a)- 
(d), of any foreign application(s) for patent or inventor's certificate listed below and have also 
identified below any foreign application for patent or inventor's certificate having a filing date 
before that of the application on which priority is claimed: 

Priority 

Prior Foreign Applicatlon(s) Claimed 



(Number) (Country) (Day/MonthA'ear Filed) Yes No 



(Number) (Country) .(Day/MonthA'ear Filed) Yes No 



(Number) (Country) (Day/MonthA^ear Filed) Yes No 



I hereby claim the benefit under title 35, United States Code, Section 119(e) of any United 
States provisional application(s) listed below 

6CM)71.566 January 15. ig9R 

(Application Number) Filing Date 



(Application Number) Filing Date 

I hereby claim the benefit under Title 35, United States Code, Section 120 of any United States 
application(s) listed below and, insofar as the subject matter of each of the claims of this 
application is not disclosed in the prior United States application in the manner provided by the 
first paragraph of Title 35, United States Code, Section 112, I acknowledge the duty to disclose 
all information known to me to be material to patentability as defined in Title 37, Code of 
Federal Regulations, Section 1.56 which became available between the filing date of the prior 
application and the national or PCT international filing date of this application: 



{Application Number) 


Filing Date 


(Status 


" patented, 








pending, abandoned) 


(Application Number) 


Filing Date 


(Status 


- patented, 








pending, abandoned) 



I hereby appoint Aloysius T. C. AuYeung, Reg. No. 35,432; William Thomas Babbitt, Reg. No 
39,591; Jordan Michael Becker, Reg. No. 39,602; Bradley J. Bereznak, Reg. No. 33,474- Michael 
A. Bernadicou, Reg. No. 35,934; Roger W. Blakely, Jr., Reg. No. 25,831; Gregory D. Caldwell 
Reg. No. 39,926; Kent M. Chen, Reg. No. 39,630; Lawrence M. Cho, Reg. No. 39,942; Thomas ' 
M. Coester, Reg. No. 39,637; Roland B. Cortes, Reg. No. 39,152; William Donald Davis, Reg No 
38,428; Michael Anthony DeSanctis, Reg. No. 39,957; Daniel M. De Vos. Reg. No. 37,813- 
Tarek N. Fahmi, Reg. No. 41,402; James Y, Go, Reg. No. 40,621; Sharmini Nathan Green, Reg 
No. 41,410; David R. Halvorson, Reg. No. 33,395; Eric Ho, Reg. No. 39,711; George W Hoover 
II, Reg. No. 32,992; EricS. Hyman, Reg. No. 30,139; Dag H. Johansen, Reg. No. 36.172- 
Stephen L. King, Reg. No. 19,180; Michael J. Mallie, Reg. No. 36,591; Kimberley G. Nobles 
Reg. No. 38,255; Ronald W. Reagin, Reg. No. 20,340; James H. Salter, Reg. No. 35,668- 
William W. Schaal, Reg. No. 39,018; James C. Scheller, Reg. No. 31,195; Charles E. 
Shemwell, Reg. No. 40,171; Maria McCormack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff 
Reg. No. 25,128; Allan T. Sponseller, Reg. No. 38,318; Steven R. Sponseller, Reg. No 39 384- 
Judith A. Szepesi, Reg. No. 39,393; Edwin H. Taylor. Reg. No. 25,129; George G. C. Tseng,' Reg 
No. 41,355; Lester J. Vincent, Reg. No. 31,460; John Patrick Ward, Reg. No. 40,216- Ben 
J. Yorks, Reg. No. 33,609; and Norman Zafman, Reg. No. 26,250; my attorneys; and Robert 
Andrew Diehl, Reg. No. 40,992; Thomas A. Hassing, Reg. No. 36,159; and Edwin A. Sloane Reg 
No. 34,728; my patent agents, of BLAKELY, SOKOLOFF. TAYLOR & ZAFMAN, with offices looted 
at 12400 Wilshire Boulevard, 7th Floor, Los Angeles. California 90025, telephone (310) 
207-3800; and James R. Thein, Reg. No. 31,710, my patent attorney; with full power of 
substitution and revocation, to prosecute this application and to transact all business in the 
Patent and Trademark Office connected herewith. I also hereby appoint Mark Aaker Reg No 
32.667, Paul D. Carmichael, Reg. No. 18,679; Richard Liu, Reg. No. 34,377; Helene Plotka 
Workman, Reg. No. 35,981; Edward W. Scott, IV, Reg. No. 36,000; and Nancy R. Simon Reg 
No. 36,930; my attomeys; of APPLE COMPUTER, INC., located at 1 Infinite Loop, MS- 38-PAT 
Cupertino, California 95014, telephone (408)974-9453, with full power of substitution and 
revocation, to prosecute this application and to transact all business in the Patent and 
Trademark Office connected herewith. 



I hereby declare that ait statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize the validity of the application 
or any patent issued thereon. 

Full Name of Sole/First Inventor Anne Jones 

Inventor's Signature Civ--^ 



5^T^ Date 1 in 



Residence Redwood Cttv. Califomia Citizenship United States 

(City, State) (Country) 

Post Office Address 3817 Hamilton Way 

Redwood Citv. CA 94062 



Full Name of Second/Joint Inventor Jay Geagan 



Inventor's Signature \\ Date / ^ 1"^ H / / 7 

Residence San Jos e. Cafrfomia Citizenship United States 

(City, State) (Country) 



Post Office Address 5475 Prospect Road #212 



San Jose. CA 95129 



Full Name of Third/Joint Inventor Kevin L. Gong 

inventor's Signature . _ Date^ 

Residence Sunnyvale. Califomia Citizenship United States 

(City, State) (Country) 

Post Office Address 955 Escalon Avenue #515 

Sunnwale. CA 94086 



Full Name of Fourth/Joint Inventor Alagu Periyannan 



Inventor's Signature F ■A(i>q>'S-~J.i^^ Date "7 /'^/ 

Residence San Francisco. Califomia Citizenship India 

(City, State) (Country) 

Post Office Address 74 Everglade Drive 

San Francisco. CA 94132 
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Full Name of Fifth/Joint Inventor David W. Singer 



Inventor's Signature ^^^T? Date ]A^^ ^ >(' 

Residence San Francis co. California Citizenship United Kingdom 

(City, State) (Country) 

Post Office Address 268 Wawona Street 

San Francisco. CA 94127 
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Title 37, Code of Federal Regulations, Section 1.56 
Duty to Disclose Information Material to Patentability 



(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office Is aware of and evaluates the teachings of all infomriation material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in dealing 
with the Office, which Includes a duty to disclose to the Office all information known to that Individual to be 
materia! to patentability as defined in this section. The duty to disclosure infomnation exists with respect to 
each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Infomnation material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the infomnation is not material to the patentability of any claim remaining 
under consideration in the application. There is no duty to submit Infomnation which is not material to the 
patentability of any existing claim. The duty to disclosure all information known to be material to patentability 
is deemed to be satisfied if all Infomnation known to be material to patentability of any claim issued in a patent 
was cited by the Office or submitted to the Office in the manner prescribed by §§1.97{b)-(d) and 1.98. 
However, no patent will be granted on an application in connection with which fraud on the Office was practiced 
or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. The Office 
encourages applicants to carefully examine: 

(1 ) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, infomnation is material to patentability when It is not cumulative to 
infomnation already of record or being made or record in the application, and 

(1 ) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) it refutes, or Is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the Information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each temn in the claim Its 
broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1 ) Each Inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively Involved in the preparation or prosecution of the 
application and who is associated with the Inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or Inventor may comply with this section by 
disclosing infomnation to the attorney, agent, or inventor 



